Effective Management of High Volume Numeric Data with Histograms

How do you capture and organize billions of measurements per second so that:

• You can answer a rich set of queries effectively (percentiles, counts below X, aggregations across streams)
• You don’t blow through your AWS budget in minutes

In this talk, we'll take a look at both log-linear and cumulative histograms and how they provide advantages over storing data as quantiles, averages, and other histogram implementations such as linear and fixed-bucket. An open source histogram software library will be explained, and sample statistical operations using it will be shown. You'll come away with an understanding of use histograms to make your data engineering life easier.