Extract - Tiered Transform - Load (ETTL): A pipeline for a modular, scalable, and observable Internal Analytics platform

Dealing with diverse and constantly evolving data sources is a challenge. Now let’s add security and compliance requirements, as well as different levels of sensitivity in your data model. Finally mix in a pipeline that has to remain easy to iterate on, facilitate backfill of historical data, and have backups. It can quickly become a nightmare.

Here is how a small team of data engineers tackled these while maintaining a modular, scalable, and observable Internal Analytics platform, using a tiered “Transform” step in a pipeline where Luigi is the maestro and Spark the hard worker.