Infiniband

Free Silverstorm 9024 SDR (single data rate, 10Gbits/sec) + cheap eBay Infiniband HCAs (host channel adapters) + somewhat less cheap cabling = fast networking.

Linux seems to have support for just every HCA that sold in any volume (which should not be surprising, really). Some friends who have worked in Infiniband connected supercomputing clusters have recommended staying with Mellanox HCAs. Depending on one's patience, whether one or two ports are desired on the card, if you're looking for SDR (10Gbps), DDR (20Gbps), or QDR (40Gbps), the phase of the moon (a stand-in for eBay sellers' mood), these HCAs can be found for under $20US including shipping. As this is written, I've acquired one Mellanox MHES-18XT and two Mellanox MHGA28-XTC HCAs.

Mellanox has an Infiniband stack available for VMware ESXi, but it appears to only support newer cards than mine.

Firmware updates for MHES-18XT

I did this, but neglected to keep any notes figuring I'd be getting more soon and writing it down then. I suck.

Firmware updates for MHGA28-XTC

ib_mthca driver tells me that

[   12.182088] ib_mthca 0000:0d:00.0: HCA FW version 5.2.916 is old (5.3.000 is current).
[   12.197980] ib_mthca 0000:0d:00.0: If you have problems, try updating your HCA FW.

So that's nice, but how? Google to the rescue...