Musings on project management: Vanity metrics

Wednesday, November 5, 2014

Vanity metrics

Vanity Metrics: Actually, until recently, I had not heard of vanity metrics, aka VM. Now, I am writing about them! Does that make me a VM SME?...

So, some definition, as given to us by VM inventor Eric Ries, as posted at fourhourworkweek.com

The only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions. Unfortunately, the majority of data available in off-the-shelf analytics packages are what I call Vanity Metrics. They might make you feel good, but they don’t offer clear guidance for what to do.

So, some examples -- as cited my Mike Cohn in an email blast about Reis' ideas:

Eric Ries first defined vanity metrics in his landmark book, The Lean Startup. Ries says vanity metrics are the ones that most startups are judged by—things like page views, number of registered users, account activations, and things like that.

So, what's wrong with this stuff? VMs are not actionable.. that's what's wrong. The no-VM crowd says that a clear cause-and-effect relationship is not discernible, and thus what action (cause) would you take to drive the metric higher (effect)? Well, you can't tell because there could be many cause, some indirect, that might have an effect -- or might not. The effect may be coming from somewhere else entirely. So, why waste time looking at VMs if you can't do anything about it?

Ries goes on to tell us it's all about "actionable metrics", not vanity metrics. AMs are metrics with a direct cause and effect. He gives some examples:

Split tests: A/B experiments produce the most actionable of all metrics, because they explicitly refute or confirm a specific hypothesis
Per-customer: Vanity metrics tend to take our attention away from this reality by focusing our attention on abstract groups and concepts. Instead, take a look at data that is happening on a per-customer or per-segment basis to confirm a specific hypothesis
Cohort and funnel analysis: The best kind of per-customer metrics to use for ongoing decision making are cohort metrics. For example, consider an ecommerce product that has a couple of key customer lifecycle events: registering for the product, signing up for the free trial, using the product, and becoming a paying customer

Now, it's time to introduce my oft cited advice: Don't confuse -- which is actually easier to write than to do -- cause-effect (causation) with correlation (coordinated movements, but not causation)

Causation: because you do X, I am compelled (or ordered, or mandated) to do Y; or, Y is a direct and only outcome of X. I sell one of my books (see below the books I wrote that you can buy) and the publisher sends me a dollar ninety-eight. Direct cause and effect; no ambiguity. Actionable: sell more books; get more money from the publisher.
Correlation: when you do X, I'll be doing Y because I feel like doing Y, but I could easily choose not to do Y, or choose to do Z. I might even do Y when you are not doing X. Thus, the correlation of Y with X is not 100%, but some lesser figure which we call the coefficient, typically "r". "r" is that part of the Y thing that is influenced consistently by X

So, what is the actionable thing to do re X if I want you to respond Y? Hard to say. Suppose "r" is only 2/3'rds. That means: 2 out of three times you'll probably respond to X with Y, but a third of the time you sit on it .... or do something else I don't care about. Bummer!

Here's my bottom line: on this blog, I watch all the VM analytics... makes me feel good, just as Ries says. But I also look at the metrics about what seems to resonate with readers, and I take action: I try to do more of the same: AM response, to be sure.

I frankly don't see the problem with having both VM and AM in the same metric system. One is nice to have and may provide some insight; one is to work on!