Bypasslink
Fast, cheap, and reliable inter-server links using repurposed Infiniband cards — and custom software to keep them honest.
Bypasslink started as an infrastructure problem. Deploying CEPH — a distributed storage system — at my-webspace.at required fast inter-node connections that standard Gigabit Ethernet couldn't deliver. Upgrading the whole network to 10 Gigabit wasn't economically viable.
The solution: secondhand Mellanox X4242A Infiniband cards, converted to Ethernet mode via system commands. Two 10 Gigabit ports per card. In a three-server setup, each machine can be directly linked to the others peer-to-peer.
The remaining problem was routing. Configuring routing tables for Infiniband prioritisation proved insufficient — Linux couldn't intelligently bypass failed connections. We wrote custom software that continuously monitors connectivity and dynamically reconfigures routing tables, failing over to slower interfaces within two seconds when problems occur.
Side note: the cards required 3D-printed slot covers for full-height PCIe compatibility. Designed in OpenSCAD, printed on a Renkforce RF100.