Lab

Where the nanoseconds go.

I once spent two weeks shaving microseconds off a hot path. A competitor beat us anyway, with a shorter cable and a better switch. This scale is why: eight orders of magnitude, from a cache hit to the microwave link between Chicago and New Jersey, and every value links to its source.

Show
1 µs Light in fibre covers about 200 m. 9 of 20 are faster. Next, at 1.38 µs: Fix8, encoding a NewOrderSingle.

Drag the cursor or use the arrow keys

  1. L1 cache hit ~1 ns Big-O Was Never the Bottleneck. The Cache Line Was. (opens in a new tab)
  2. Arista 7130, port to port as low as 4 ns Arista 7130 Series (opens in a new tab)
  3. One metre of fibre ~5 ns ITU-T G.114, one-way transmission time (opens in a new tab)
  4. STAC-T0 record, FPGA tick-to-trade 13.9 ns STAC-T0 on AMD Alveo UL3524 (2024) (opens in a new tab)
  5. Arista MetaMux, servers to the exchange 39 ns on average Arista 7130 MetaMux (opens in a new tab)
  6. Disruptor, one hop 52 ns LMAX Disruptor technical paper (2011) (opens in a new tab)
  7. Main memory reference ~100 ns Big-O Was Never the Bottleneck. The Cache Line Was. (opens in a new tab)
  8. Arista MetaProtect, filtering 135 ns Arista 7130 MetaProtect (opens in a new tab)
  9. Hardware tick-to-trade, Exegy nxAccess 389 ns Exegy nxAccess (opens in a new tab)
  10. Fix8, encoding a NewOrderSingle 1.38 µs Fix8 (opens in a new tab)
  11. Software tick-to-trade, Exegy nxAccess 1.5 µs Exegy nxAccess (opens in a new tab)
  12. Kernel bypass (Onload), TCP half round trip 3.1 µs Arista and Solarflare low-latency white paper (2011) (opens in a new tab)
  13. Next-generation firewalls, UDP at 90% load 7–177 µs NSS Labs, NGFW performance report (2018) (opens in a new tab)
  14. Linux kernel TCP stack, half round trip 8.1 µs Arista and Solarflare low-latency white paper (2011) (opens in a new tab)
  15. ArrayBlockingQueue, one hop 32,757 ns LMAX Disruptor technical paper (2011) (opens in a new tab)
  16. IEX speed bump, coiled fibre 350 µs IEX technology (opens in a new tab)
  17. ZGC pause, at most 1 ms OpenJDK ZGC wiki (opens in a new tab)
  18. Light, Chicago to New Jersey 3.93 ms Laughlin, Aguirre and Grundfest (2013) (opens in a new tab)
  19. Microwave, Chicago to New Jersey 3.982 ms Quincy Data and McKay Brothers (2016) (opens in a new tab)
  20. Treasury futures and cash, lead–lag offset ±5 ms Dobrev and Schaumburg, Liberty Street Economics (2015) (opens in a new tab)

Every category is switched off.

A dot is a single value, a bar a range. Each figure is as its source states it: datasheets, benchmark reports, papers. The argument around them is in two pieces of The Converging Line: the cache line (opens in a new tab) and the cable (opens in a new tab).