Why metrics should be a first class citizen in a modern data stack

Before starting Falkon, I interviewed 46 data and analytics teams to understand their day to day workflows. One of the most important things I learned was how much confusion exists about what metrics are.

“We have lots of dashboards to do reporting. Those are our metrics.”

This was a common and scary thing I heard repeatedly.

Metrics aren’t just dashboards. They are the foundation of an OKR driven company and without them "test and learn" is guesswork at best.  

Revenue — a metric we all understand

Before getting into the generalized definition of a metric, let’s explore Revenue — a metric we all mostly treat like a first class citizen.

1. Revenue represents the ultimate performance indicator.
2. Revenue is tracked over time.
3. A company defines revenue in one specific way.
4. This definition does not change frequently.
5. When it does change, it’s with serious deliberation and communication.
6. Finance teams have governance for revenue data and workflows.
7. The impact of a company’s initiatives are judged based on their impact to revenue.
8. Revenue is not just for the finance team, but rather a company wide concept since it is impacted by activities from many different teams.

Now contrast this to Daily Active Users (DAU), another KPI, and things start to get higgledy piggledy.

Often, we find:

1. DAU is defined differently by each team within a company.
2. Even within one team, the definition of DAU is updated without much discussion or communication.
3. There’s little to no governance on the data that goes into computing DAU, nor its definition.
4. Teams guess the impact of their activities on DAU because it’s a moving target. I’d classify all experimentation efforts as guesswork if they’re building upon shaky metrics.
5. Given the lack of standardization, cross team collaboration becomes inordinately difficult.

Purpose

A metric is a quantifiable proxy for a performance indicator that an organization cares about for the long haul. Daily Active Users is a metric for the HBO app. HBO probably cares about tracking this key performance indicator of customer health over time. 

In contrast, the answer to a question like “How many users watched the first episode of The Watchmen within the first week?” is not a metric. We should be asking such questions of our data but they don’t need to be promoted to “metric” status.

Time

A metric has one mandatory dimension and that is time. Simply put, a metric is a unit of measurement that is tracked over a durable period of time. Internally, this is stored as time series data. Often, the more granular the time, the better. 

Definition stability

A metric’s definition is stable and does not change often. Why? Because if a metric’s purpose is to be a key performance indicator, and it’s definition keeps changing, it’s impossible to measure the impact of organizational activities. Did Conversion Rate go up because of a test we just rolled out or because we updated our definition of the denominator from Sessions to Distinct Users?

Dimensions

n a metric (i.e. the criteria you can slice and dice by) change a lot, especially in dynamic and growing businesses. New Product SKUs, new regions etc. are added so we can look at Conversion Rate for these new SKUs and regions. However the mathematical definition of the metric remains unchanged.

This is similar to physical fitness. The KPI definitions of fitness don’t change often, but the tactics to achieve them do and should.

Governance

The process of defining and updating a metric is more like code, and less like ad hoc querying. Just as with code, metric definitions are source controlled, versioned, go through a peer review process, allow for rollbacks in case of erroneous updates, and have strong lineage.

Why should we care?

It’s a prerequisite to being data driven

As modern business and data leaders, we want to be data driven. To support this ambition, we’ve built out our data warehouse foundation. We’ve invested in a good experimentation platform. We’ve taken the OKR training and go through OKR planning every quarter. We have our Monday weekly business reviews (WBRs) to track our performance. We’ve even built out our Looker and Tableau dashboards for these meetings. But without a foundational investment in metrics, there’s a critical break in the stack that’s likely to result in garbage in → garbage out.

It’s hard to collaborate without metrics

Metrics are impacted by company wide initiatives. For instance, Conversion Rate is impacted by the traffic the marketing team generates, the user experience the product team provides, and the pricing the business team determines. Without a shared definition of a metric like Conversion Rate, team collaboration becomes inordinately difficult.

This is further exacerbated by having metric definitions locked up in a proprietary tool or BI vendor. It means metrics cannot be shared or used in any other application. For instance if a data scientist wants to perform correlation analysis between Conversion Rate and some exogenous factors (like the weather), they are unable to get the metric out of the tool it’s locked up in to perform this analysis.

What’s next?

In the next article, Aakash (my co-founder, our CTO, and resident dad-joke expert) will share what we’ve learned about building a modern metric system that truly makes them a first class citizen in the modern data stack.

Want more articles like this? Follow Falkon on LinkedIn >