Evolving Data Pipelines at Scale

When it comes to developing new data pipelines and evolving existing ones, the available tooling lags behind what's available to software developers. Creating new development environments is cumbersome, populating these environments with representative data is compute-intensive, and the process of deploying changes to production is ad-hoc and error-prone. All of these factors drive up development costs, slow down iteration speed, and make data less reliable.

At the core of the open-source project SQLMesh is a novel development workflow, born from our collective experience at companies like Airbnb, Apple, Google, and Netflix. It is designed to handle the complexities of evolving data pipelines at an internet scale. In this talk, we’ll discuss challenges faced by data practitioners today and highlight core SQLMesh concepts addressing them, including virtual data environments, automatic change categorization, continuous testing, and accurate change previews.