While experimentation gives us clean effect measures, connecting those results to real-world business decisions is messy. In this talk, I’ll walk through two case studies at Roblox that highlight this challenge and explore some causal inference methods to help bridge the gap. The first focuses on attributing observed year-over-year business growth to product launches. The strategic need here is twofold: to understand how much of our growth is driven by the innovations we shipped, and to reconcile different measurements of business performance— experiment results and long-term growth trends—into a coherent narrative. The core challenge is isolating product impact from organic growth (in the absence of these launches) in the topline metrics we observe.The second case study addresses how to generalize A/B test results to a broader population, without requiring an explicit evaluation of covariate shift between the experiment and target population—making the approach scalable across experiments and surfaces. This framing is essential for fair comparisons across product areas that vary in reach and in how amenable they are to metric movement, enabling more effective prioritization across teams.Together, these cases reflect a broader goal: building a common measurement language that connects local experimental results to global business impact—so organizations can make more strategic, data-informed decisions.
