Creating our own Kubernetes and Docker to run our data infrastructure

When Erik started working on an infrastructure solution for data teams, he quickly realized that Docker and Kubernetes were too slow and not right for the use case. Almost everyone said it was a bad idea to build their own system, but they ended up building it anyway. In this talk, Erik will share how Modal starts 1000s of large containers in seconds, and what they had to do under the surface to build this. This includes a custom file system written in Rust, their own container runtime, and their own container image builder. This talk will give you an idea of how containers work along with some of the low-level Linux details underneath. But along the way we'll also talk about many infrastructure tools hold data teams back, and why they deserve faster and better tools.