How percentile approximation works and why it's more useful than averages (blog.timescale.com)

Article summary

The article discusses the importance of using percentiles instead of averages when analyzing data, particularly in scenarios like API response time monitoring. Percentiles provide a more accurate representation of the data distribution, whereas averages can be skewed by outliers. The article also introduces the concept of percentile approximation and its implementation in TimescaleDB hyperfunctions. This allows for efficient computation of percentiles on large datasets.

Main themes

• percentile approximation
• data distribution analysis
• algorithm choice
• streaming data
• memory constraints
• response time monitoring
• system performance optimization

What commenters say

• Using percentiles is more effective than using averages for understanding data distributions, especially in the presence of outliers.
• The choice of algorithm for calculating percentiles depends on the specific use case and requirements, such as memory constraints or streaming data.
• Some algorithms, like T-Digest and UDDSketch, provide efficient and accurate percentile calculations, but may have limitations, such as non-determinism or difficulty in merging summaries.
• The minheap-maxheap approach can be used for streaming data to estimate the median, but it may not be suitable for large datasets that do not fit in memory.
• QuickSelect and median-of-medians algorithms can be used to find the median of a large dataset, but may have different trade-offs in terms of time and space complexity.
• The importance of limiting variation in response times is highlighted, as it can impact the accuracy of percentile calculations and the overall performance of systems.
• Different data structures and algorithms, such as CKMS quantiles and KLL algorithm, can be used for percentile calculations, each with their own strengths and weaknesses.