Data Reliability for Data Lakes

Building a modern data lake requires dealing with a lot of complexity: querying historical data + streaming data simultaneously (lambda architecture), validation to ensure data isn't too messy for data science and machine learning, reprocessing to handle failures, and ensuring ACID-compliant data updates. We created the Delta Lake project, open sourced under the Linux Foundation, to relieve data scientists and data engineers from these complex systems problems and instead enable them to focus on extracting value from data. In this talk, we'll dive into these challenges and how ACID transactions solve them. We'll discuss patterns that emerge when you can focus on data quality and the nitty gritty internals of ACID on Spark which enable this focus.