Building A Scalable Financial Data Store

Often as data engineers, we’re trusted with providing a “source of truth” for downstream consumers. We’re on the hook to provide accurate, timely, sensible information for people across the organization. Today we’re going to talk about the most crucial data of all — money. We’re going to discuss handling financial data from the perspective of a B2B company.

There are many different consumers of financial data within a company. Let’s go through a typical org. Your Business Development team’s bonuses are determined by the amount of revenue they generate. Your analytics team needs to understand how your company funnels user engagement into revenue, and, last and most importantly, your external users expect you to provide accurate and simple access to their orders.

Building a data service with so many downstream stakeholders creates unique challenges. How do you build a system robust enough to handle the requirements of each party, and how do you prioritize their differing interests?

In this talk, we’ll discuss design strategies (such as immutability, optimizing ease of aggregation, hybrid OLTP/OLAP approach) that enable you to design and build data models and interfaces that can flexibly serve variant user needs while fulfilling high accuracy requirements. Lastly, we’ll talk about some strategies for scaling and optimizations.