Five questions to ask about any file transfer benchmark

Vendor benchmarks, ours included, are only useful if you can tell what was measured. Five questions that separate a result you can reason about from a number on a slide, and how our own published test answers them.

RIPTON CLOUD team, Product and engineering · · 2 min read

In short

  • A benchmark without RTT and loss is a benchmark of a room, not a network.
  • Ask what the baseline was, what 'done' meant, how many files, and whether you could reproduce it.
  • Our published test states 150 ms and 3% loss, a TCP baseline, completion as verified bytes on disk, and both a single 1 GB file and 100,000 small files.

The five questions

  1. 1What were the RTT and the loss rate? Without them the number describes the test room. A result at 1 ms RTT tells you nothing about a route to another continent, because the thing that limits long-distance transfers is absent.
  2. 2What was the baseline? 'Faster' needs a denominator. A tuned parallel-stream TCP tool, a single curl, and a consumer sharing site are different baselines and produce different multiples for the same product.
  3. 3What did 'done' mean? A transfer is complete when the bytes are on disk at the destination and verified. Measuring to the last packet sent, or excluding packaging and verification, shortens the time in ways that do not show up in the headline.
  4. 4How many files? One 10 GB file and 100,000 small files that add up to 50 MB are different workloads. Per-file overhead dominates the second and is invisible in the first. A benchmark that only shows one of them has only tested half the problem.
  5. 5Could you reproduce it? Instance types, tool versions, commands and the way loss and latency were introduced should be stated. If you cannot run it, you cannot check it, and you cannot compare it with your own route.

How our published test answers them

QuestionOur published EC2 test
RTT and lossAbout 150 ms round trip and 3% packet loss, introduced on the path between two instances.
BaselineA TCP transfer of the same payload over the same impaired path.
DoneBytes on disk at the receiver and verified.
FilesTwo workloads: one 1 GB file, and 100,000 files of 512 bytes each.
Result1 GB in about 4 minutes against about 30 for the baseline. 100,000 small files in 0.98 seconds against 2 minutes 52 seconds.

We publish the multiple against the TCP baseline, 7.5x for the single file, because the baseline and conditions are stated. We do not publish multiples against named products, because we would have to test their tools under conditions they did not choose, and you would have no way to check we had configured them fairly. If you want that comparison, run it yourself on your route with both tools; the proof-of-concept page describes exactly that.

Related

More from the blog