Before starting Falkon, I interviewed 46 data and analytics teams to understand their day to day workflows. One of the most important things I learned was how much confusion exists about what metrics are.
“We have lots of dashboards to do reporting. Those are our metrics.”
This was a common and scary thing I heard repeatedly.
Metrics aren’t just dashboards. They are the foundation of an OKR driven company and without them "test and learn" is guesswork at best.
Before getting into the generalized definition of a metric, let’s explore Revenue — a metric we all mostly treat like a first class citizen.
Now contrast this to Daily Active Users (DAU), another KPI, and things start to get higgledy piggledy.
Often, we find:
A metric is a quantifiable proxy for a performance indicator that an organization cares about for the long haul. Daily Active Users is a metric for the HBO app. HBO probably cares about tracking this key performance indicator of customer health over time.
In contrast, the answer to a question like “How many users watched the first episode of The Watchmen within the first week?” is not a metric. We should be asking such questions of our data but they don’t need to be promoted to “metric” status.
A metric has one mandatory dimension and that is time. Simply put, a metric is a unit of measurement that is tracked over a durable period of time. Internally, this is stored as time series data. Often, the more granular the time, the better.
A metric’s definition is stable and does not change often. Why? Because if a metric’s purpose is to be a key performance indicator, and it’s definition keeps changing, it’s impossible to measure the impact of organizational activities. Did Conversion Rate go up because of a test we just rolled out or because we updated our definition of the denominator from Sessions to Distinct Users?
n a metric (i.e. the criteria you can slice and dice by) change a lot, especially in dynamic and growing businesses. New Product SKUs, new regions etc. are added so we can look at Conversion Rate for these new SKUs and regions. However the mathematical definition of the metric remains unchanged.
This is similar to physical fitness. The KPI definitions of fitness don’t change often, but the tactics to achieve them do and should.
The process of defining and updating a metric is more like code, and less like ad hoc querying. Just as with code, metric definitions are source controlled, versioned, go through a peer review process, allow for rollbacks in case of erroneous updates, and have strong lineage.
It’s a prerequisite to being data driven
As modern business and data leaders, we want to be data driven. To support this ambition, we’ve built out our data warehouse foundation. We’ve invested in a good experimentation platform. We’ve taken the OKR training and go through OKR planning every quarter. We have our Monday weekly business reviews (WBRs) to track our performance. We’ve even built out our Looker and Tableau dashboards for these meetings. But without a foundational investment in metrics, there’s a critical break in the stack that’s likely to result in garbage in → garbage out.
It’s hard to collaborate without metrics
Metrics are impacted by company wide initiatives. For instance, Conversion Rate is impacted by the traffic the marketing team generates, the user experience the product team provides, and the pricing the business team determines. Without a shared definition of a metric like Conversion Rate, team collaboration becomes inordinately difficult.
This is further exacerbated by having metric definitions locked up in a proprietary tool or BI vendor. It means metrics cannot be shared or used in any other application. For instance if a data scientist wants to perform correlation analysis between Conversion Rate and some exogenous factors (like the weather), they are unable to get the metric out of the tool it’s locked up in to perform this analysis.
In the next article, Aakash (my co-founder, our CTO, and resident dad-joke expert) will share what we’ve learned about building a modern metric system that truly makes them a first class citizen in the modern data stack.
Want more articles like this? Follow Falkon on LinkedIn >