Measurement: Four Assumptions to Live By

We are our own worst enemy when it comes to measurement. There are literal text books on the many ways we self-sabotage our measurement efforts. But there is good news.

The most common error we make is also the easiest to solve, and that’s what I want to talk discuss today.

Not Measuring

Yep, the most common measurement failure isn’t forgetting to “carry the 1”, its choosing not to measure anything at all.

It seems strange that in our data-centric world we’d still see so many organizations choose not to even try to measure the strategic problems or solutions they invest millions of dollars into. The excuses are not new or unique however. If you work in an analytics or improvement role you’re probably familiar with many of them. Maybe you’ve used a few of them (I certainly have). Here are some of my favorite:

• “Our problem is too unique and difficult to lend itself to measurement”
• “There’s no way to quantify that.” (said about squishy terms like “risk”, “satisfaction”, or “performance”)
• “We don’t have the data available” (engineering won’t share the data, system A and system B don’t use the same code base, etc.)
• “It’d take way too much effort to get statistically significant results.”

Four Measurement Assumptions

Take a leap of faith, and start with four assumptions about your current measurement problem.

Why? I’ve found that in every measurement challenge I’ve faced, they all have been true. And using them as a starting point will help you avoid most of the dead-end excuses I mentioned above.

1. Everything has been measured before.

I’d love to give you a metric about the likelihood that the thing you are trying to measure is truly unique to you or your company. But it’d be both made up – and an understatement.

Don’t reinvent the wheel, if you’re feeling stuck then do a bit of research and see how other companies or industries have solved your problem. I once had a middle manager tell me, with a straight face, that there simply was no way to measure or predict the type of risk her team dealt with. It was chaos, and that was that.

Turns out there is a whole industry based on measuring the likelihood of random bad things happening however (car insurance anyone?). With a little effort, we were able to find a methodology that we could almost cut and paste to fit her exact scenario.

2. You have more data than you think.

Despite the deluge of data at our fingertips, many organizations are still starving for useful information. We can sometimes exacerbate this by limiting what we qualify as “data”. Sure, maybe your ancient CRM software doesn’t have functionality to export key data fields you need to perform a client analysis, but what other sources of information may you have (both direct and indirect)?

3. You need far fewer data points than you think.

We often get stuck on an imaginary threshold of certainty, below which data has no value. The common arm-chair statistician retort “yes, but is it statistically significant?” betrays this misunderstanding of the purpose of measuring things. When we choose to measure something, most of the time we don’t need (nor is it possible to get) an exact answer.

Yes, we definitely need to make sure we understand how our data was gathered and ensure we’re making valid conclusions. At it’s heart however, measurement is really just about reducing uncertainty. If you’re starting from a place of very high uncertainty, then a very small amount of information can actually be extraordinarily beneficial (look up the rule of five, to see how you can make a 93.75% accurate prediction with just five data points).

4. New observations are more accessible than you think.

Lastly, the act of gathering data can often seem like an entire project unto itself, and it definitely can be. But, if we unhinge ourselves from perfection – we’ll find there is actually a great deal of data gathering we can do with very little effort. Sending an email to a few customers asking for feedback on a recent interaction may not be worthy of publication in a scientific journal, but it can still give you meaningful information about customer perception and pain-points (particularly if you previously were just guessing).

If you use these four assumptions as a starting place, you’ll find that the most common objections to measurement will melt away before your analytical stare. The end measurement or method may not exactly resemble what you or your team thought, but you WILL have added real value by reducing the uncertainty of whatever decision or project you are involved in.

Happy Measuring!

If you liked these four measurement assumptions, check out “How to Measure Anything” by Douglas Hubbard. Its my favorite data science book and has loads more great tips like this.