Intelligent Orchestration - Data's Missing Link

We have seen great advancements and innovations across the data management landscape - from large-scale processing engines to cloud-based data warehouses and even out to BI and machine learning tools. But why is it that the most powerful systems of today are being controlled and orchestrated by some of the most rudimentary technologies? How data moves and is transformed is still dictated by manual, hard-coded triggers and rules, resulting in slow development cycles and brittle pipelines. For many data engineering teams, this also causes a high maintenance burden and disproportionate amount of time spent combing through code and logs, and constantly tuning parameters just to keep things running. As the amount of pipelines and dependencies scale, these challenges compound and eclipse what can be manually constructed and managed by even the most talented data engineering teams.

This is what led us to rethink the state of pipeline orchestration. In taking a page from other technologies, such as Kubernetes, React, and even compilers, we opted to move away from imperative, task-based design and move to declarative, data-centric automation. In this session, Sean Knapp, CEO & founder of Ascend, will discuss what led us to take this approach and essentially rearchitect pipeline orchestration from the ground up. In particular, he will talk through:

- Current assessment of orchestration tools and advancements

- Tradeoffs and design decisions in moving from task-centric to data-centric architectures

- Challenges Ascend faced running a massive-scale, elastic infrastructure built on Kubernetes and Apache Spark

- How Ascend built a Control Plane to combine declarative configurations and automation to power intelligent orchestration.