Emerging Data Roles: The Analytics Engineer

Analytics Engineer: this term has started showing up in blog posts and job listings. It all happened quickly; just a couple of years ago, it wasn't a thing our friends in the data ecosystem talked about. So how did it start trending, what is it exactly, and is it here to stay? We decided to take a closer look, and here's what we found out.

Naming the Need

At present, there's an unmissable connection between the 'Analytics Engineer' term and the ecosystem around dbt, the command line data transformation tool developed by Fishtown Analytics. However, the company's Head of Marketing Janessa Lantz told us that they wouldn't take credit for coining it: 

"It started popping up in the dbt community in 2018. Michael Kaminsky was probably the first to formally write anything about it when he published [his] post in early 2019."

Janessa is referring to "The Analytics Engineer", a post published by Locally Optimistic and authored by Michael Kaminsky, currently a consultant, and formerly a Director of Analytics at Harry's. As Michael recalled in a recent Locally Optimistic webinar discussing his blog post, not many were using that term at the time, but the need for it had started to emerge:  

"I realized pretty early on that we had this need for an analytics engineer, […] who would be on the analytics team working with us, pursuing our goals, and not necessarily being on the data engineering team, but [who still] had software engineering skills. And it was a real battle with my organization to actually be able to hire for that role because no one had ever heard of it before. […] So after going through that experience and talking with lots of other people who were having similar experiences, I wanted to write this blog post about why this role is so important and what it actually means."

Well, what it actually means is still somewhat open for debate – and we'll come back to it – but Michael's seminal post took a pretty good stab at coming up with a general definition:

"The analytics engineer sits at the intersection of the skill sets of data scientists, analysts, and data engineers.They bring a formal and rigorous software engineering practice to the efforts of analysts and data scientists, and they bring an analytical and business-outcomes mindset to the efforts of data engineering."

An Epiphany

One year later, it is fair to say that the Locally Optimistic post resonated deeply with those who were already doing analytics engineering, even when they weren't calling it that. In the above-mentioned webinar, dbt Community Manager Claire Carroll recalls how she felt at the time: 

"I remember reading Michael's post in January of 2019 and feeling very seen. I had given a presentation six months earlier where I wrote my title as "Data Something" […] so as soon as I saw that analytics engineer title I was like, ‛yeah, that's what I am!’"

Claire also wrote her own post about this topic, in which she shares a key point made by Michael and several other people we talked to: the evolution of data tools has played a key role in this shift. While she still recalls a time when Excel was her primary tool, that changed quickly, and the analytics engineering toolkit is now much more technical and multifaceted.

Dylan Baker is well-placed to know this: he started doing what ended up being analytics engineering work at Simba Sleep and Growth Street, before starting his analytics engineering consultancy. The tools he uses the most right now are the following:

• Snowflake, BigQuery and Redshift for data warehouses;
• Fivetran and Stitch for data extraction and loading;
• dbt for the transformation layer;
• Looker on the BI/modeling side.

Others use similar stacks with small variations, for instance with Mode or Chartio as alternatives to Looker. If you are considering which tools to use right now, we recommend reading this overview on Locally Optimistic: "Building an Analytics Stack in 2020".

One might also note, unsurprisingly, that in the same way the analytics engineering toolset is evolving, dbt's focus is also expanding, as Janessa told us: 

"We used to describe dbt as the "T" in "ETL", but it also does testing and documentation, and integrates with version control so that description is no longer comprehensive enough. Increasingly, we describe dbt as "analytics engineering software" which better captures the range of work that happens between loading data into your warehouse and analyzing it in your BI tool."

New tools have also emerged to support this workflow and integrate best practices from the software engineering world. For instance, Great Expectations focuses on fighting pipeline debt thanks to data testing and documentation; while Dylan is working on Spectacles, a command-line, continuous integration tool for Looker and LookML.

A Catalyst for Analytics

With this stack in mind, you might be wondering what differentiates an analytics engineer from a data engineer. The exact answer will vary depending on each organization, but a common denominator seems to be that analytics engineers are much more concerned with business logic and are better positioned to grasp the true implications of their work in terms of business value. 

That's also the case at Chewy, where analytics engineers differ from data engineers in terms of both skill set and responsibilities. Director of Business Intelligence at Chewy, Michael Gerstl, explained in Locally Optimistic's May 5th webinar:

"I look at our analytics engineers – my BI engineers – as jack of all trades, master of none. They're not going to be ETL engineers, but they're going to be able to write Python, they're going to be able to use Knime to build data pipelines, they're going to do data validation testing, they're going to use Git, they're going to use Tableau, and they're going to be able to sit with business users and walk with them through their problems – which is probably the biggest thing for me."

This is what makes analytics engineers an almost immediate catalyst for analytics teams, and more generally, for the businesses who hire them; and explains why they are increasingly in high demand. 

The Making of an Analytics Engineer

While there is no unique path to get into analytics engineering, Danielle Leong's career trajectory is an example of how someone can transition into this field. Although she never officially had "Analytics Engineer" as her job title, this new term very much represents the work she has been doing for the last couple of years, including in her current role at Teachers Pay Teachers. 

The transition took place during Danielle's former role at Starry, where she was hired as an analyst but ended up becoming a top contributor to the company's dbt repo. Based on this experience, she encourages recruiters to "look for a mindset, not a skill set." This can be found in workers with a large range of profiles. For instance, breaking problems down is something that professionals with a STEM education like her are familiar with, as are software engineers. 

More generally, people can go directly into analytics engineering either from a data engineering background or from an analyst one. According to Danielle, coming from the latter is probably easier, since that person will already be familiar with the business context and other 'soft' skills, which aren't the easiest to learn. 

In contrast, there is an increasing amount of resources available for an analyst to develop into an analytics engineer. In a conversation, Danielle pointed out dbt learn and dbt's jaffle shop tutorial as some of the resources she wishes existed when she got started. She thinks the fictional e-commerce store project is particularly helpful, because it gives learners a business context that they can use as a playground to learn dbt skills.

Even if this is a good substitute for direct experience, Danielle advises recruiters and managers against making previous analytics engineering knowledge a prerequisite for new hires. Instead, she recommends them to look for candidates who could grow into their roles. By doing that, companies might also end up saving money, she noted in a talk at an NYC dbt meetup last February:

"One thing that I noticed talking to a handful of companies and […] a variety of in-house recruiters and managers is that right now for someone who has some analytics engineering skills […] the price range seems to be about $120k to $130k. […] From my experience […] I would say that this is probably more of an $80k to $100k range if you want to take the opportunity to really train someone up."

Of course, your mileage may vary depending on the city you are in and the overall job market, but one thing is for sure: analytics engineers are in demand, and flexible requirements will bring more candidates.

What's Next?

With analytics engineering becoming one of the first responsibilities of new data hires, is it destined to become a career path in which people will spend decades? There are reasons to think it might not, according to Claire Carroll. Admittedly "playing devil's advocate" during Locally Optimistic's May 5th webinar, she noted that the curiosity and learning appetite of people who typically become analytics engineers might cause them to move on to other roles – especially positions that are less internally-facing and where they can more directly sense their impact.

On the other hand, Michael Kaminsky hopes that it will be more than a stepping stone, becoming instead a long-term practice at which people get better over time  –  and a crucial one for companies, to the point that analytics engineers could eventually become a bigger group than data engineers.

Obviously, it is still early to know which, if any, of these predictions is right, but the notion of "practice" seems to be key here. In our conversation, Dylan took another guess, which sounds more likely: analytics engineering might become a career for some, and a collection of things that can be leveraged by professionals in other roles. For instance, he has witnessed cases in which people who worked in marketing and sales operations and were able to write SQL made useful contributions to dbt projects.

Indeed, dbt is closely related to this democratization. When writing about the launch of the dbt Integrated Developer Environment (IDE) earlier this year, Fishtown Analytics' CEO Tristan Handy revealed that the company's focus for 2020 was to "expand the group of people practicing analytics engineering." The IDE itself is one step in that direction, and the other side is dbt’s focus on community, with a series of initiatives to connect practitioners at all stages. 

This community aspect might well hold the future of analytics engineering. By providing newcomers with the knowledge they need to learn, it will build a strong pipeline of people entering the field or dabbling in it. Perhaps equally importantly, it will give more senior people the opportunity to shine and grow their career. All of this should help analytics engineering be perceived from the outside as a valid discipline, and one that is here to stay.

Additional Resources

Here's a list of Analytics Engineering resources you might find worth exploring:

• dbt events
• dbt Learn 
• dbt Slack (by invitation)
• dbt's jaffle shop tutorial
• Locally Optimistic's blog
• Locally Optimistic's Slack (by invitation) 
• Locally Optimistic's event calendar
• Locally Optimistic's YouTube channel