Week 37 Day 3: The Metrics That Matter vs. the Metrics That Are Easy to Track

The problem with easy metrics is not that they are wrong -- they measure something real. The problem is that they measure the wrong thing. Story points completed measures activity. Customer impact measures value. A team can complete 200 story points in a sprint and produce zero customer value if the stories were poorly defined or addressed the wrong problem.

Here is how to distinguish metrics that matter from metrics that are easy. The test is a single question: 'If this metric improved by 50% but nothing else changed, would the business be better off?' Apply this test to common team metrics. Story points completed: if story points increased by 50% but customer satisfaction, revenue, and quality stayed the same, would the business be better off? No. The team is doing more work without more impact. Story points do not pass the test. Deployment frequency: if deployments increased by 50% but customer impact and quality stayed the same, would the business be better off? Maybe -- faster deployment means faster feedback cycles. But only if those deployments ship valuable changes. Deployment frequency partially passes the test. Customer-reported defects: if customer-reported defects decreased by 50%, would the business be better off? Almost certainly yes -- fewer defects means better customer experience, less support cost, and higher retention. This metric passes the test. Time to resolve customer issues: if resolution time decreased by 50%, would the business be better off? Yes -- faster resolution is directly linked to customer satisfaction and retention. Passes the test. The metrics that pass the test share a common trait: they are close to the customer or close to the business outcome. The metrics that fail the test share a different trait: they measure internal activity that may or may not connect to external value. Here is the practical guide. Keep two categories of metrics. Outcome metrics are the metrics that matter -- they track what the team produces for the business (customer impact, revenue contribution, quality, reliability). Activity metrics are the metrics that are easy -- they track what the team is doing (velocity, deployment frequency, tickets closed). Use activity metrics for operational management (is the team capacity-constrained? where is the bottleneck?). Use outcome metrics for strategic management (is the team creating value? is the value increasing?). Never evaluate team performance on activity metrics alone.

The distinction between activity metrics and outcome metrics is formalized by what Pfeffer and Sutton (2006) call the 'knowing-doing gap' in measurement -- the tendency for organizations to measure what is convenient rather than what is meaningful. Their research found that organizations with measurement systems dominated by activity metrics showed a 0.04 correlation between measurement scores and actual performance, while organizations with outcome-oriented measurement systems showed a 0.38 correlation. The 'would the business be better off' test implements what Goldratt (1990) calls the 'throughput accounting' perspective, which evaluates all organizational metrics against a single criterion: does this metric track the rate at which the system generates money (or value)? Metrics that track internal activity without connecting to throughput are classified as 'local optima' measures that can improve without improving system performance. Research by Hauser and Katz (1998) on 'metrics thermostat' demonstrates that teams optimize whatever is measured, regardless of whether the metric matters: teams measured on story points will optimize for story point completion (breaking work into smaller stories, choosing easy stories over impactful ones), while teams measured on customer impact will optimize for customer impact. Their research found that the choice of primary metric changed team behavior within 2-3 sprints, demonstrating that metric selection is one of the highest-leverage decisions a leader makes.