Uber’s Data Journey: 100+PB with Minute Latency

Uber’s mission is to provide transportation as reliable as running water, everywhere, and for everyone. To fulfil this mission, Uber relies heavily on making data-driven decisions at every level. Thus, we need to store more and more data as the business grows in addition to providing faster, more-reliable, and more-performant access to our analytical data.

The Uber data platform is built around Hadoop ecosystem and stores more than 100 PetaBytes of data. This talk will dive into our Hadoop platform journey at Uber over the past few years, where we are standing now, and what we are building next. We started by emphasizing on data reliability, solved scalability and ease-of-use challenges and are currently focusing on faster data as well as improved efficiency.We'll look behind the scene at the current technology landscape at Uber including various big data solutions like Hadoop, Spark, Hive, Presto, Kafka, Avro, and Vertica as well as Uber's open-sourced applications and services such as Hudi, Marmaray, and Peloton. We'll dive into the technical aspect of how data freshness can be reduced from 24 hours down to minutes, ease-of-use be improved by adding a Hadoop dispersal service, GDPR regulatory requirements be addressed by providing update functionality for existing append-only columnar Hadoop data, and efficiency be improved by unifying ingestion services/pipelines. You’ll leave the talk with greater insight into how things work at Uber and will be inspired to re-envision your own data platform.

In this talk we reflect on Uber’s journey with scaling our Data Infrastructure: how did we have to reinvent ourselves scaling from 1PB to 10PB to 100PB and beyond while reducing latency from 24 hours to 3h to 1h to 10 minutes, what tools did we have to make and open source to make this happen, and at what point should you think about building Data Platform.