Debunking Cloudflare’s recent performance tests (fastly.com)

Article summary

Fastly has published a response to Cloudflare's recent performance tests, claiming that the tests were flawed and misleading. Fastly argues that the tests compared a mature Cloudflare product to a beta Fastly product, and that the test methodology was biased towards Cloudflare's infrastructure. Fastly also points out that they used a free trial account for the tests, which has limited performance compared to paid accounts. The company claims that a more scientific and useful comparison would have yielded different results.

Main themes

• performance testing
• benchmarking methodology
• cloud services comparison
• programming language impact
• network latency
• test bias and fairness
• terms of service and permission requirements

What commenters say

• The programming language used for the tests, JavaScript or Rust, may have affected the results, but the difference in performance is more likely due to network latencies.
• Cloudflare's tests were misleading because they compared a beta product to a mature one, and a more fair comparison would have been between Rust on Compute@Edge and JavaScript on Cloudflare Workers.
• The test methodology used by Cloudflare was flawed, and a longer test duration or a different test setup would have yielded different results.
• Benchmarking a service without permission can be seen as prohibited, even if the terms of service require only a request for permission, as it may lead to biased results.
• Cloudflare's requirement for permission to benchmark their services can be seen as a way to avoid impartial benchmarks and potentially manipulate the results.
• The use of a free trial account for the tests limited the performance of Fastly's service and skewed the results in favor of Cloudflare.
• The choice of test locations and nodes can significantly affect the results of the performance tests, and a more diverse set of locations and nodes would have provided a more accurate comparison.