Measurement, Uncertainty and Risk
Hubbard’s definition of measurement is: “A quantitatively expressed reduction of uncertainty based on one or more observations.” If you can say, for example, that you’re 10% more confident in making a decision based on some newly acquired information, you have effectively made an observation-based measurement.
That’s assuming we know what it is we’re supposed to be measuring. Sometimes, we don’t have a clear idea of this, and defining it is the starting point of any assessment. The author will often ask clients at the outset of a project to explain what they mean by some criterion they want to measure. For example, an assignment for the Department of Veteran Affairs (VA) that called for a measurable improvement in IT security prompted him to ask the VA to define the term “IT security,” something one might consider difficult to measure. Several workshops later, they’d identified aspects such as virus attacks and unauthorized intrusions, activities whose impacts were clearly measurable.
It’s also helpful, he says, to understand why a particular measurement is being sought, so that your focus on what needs to be measured is properly directed. You can simply ask “Why do you want to measure this?”, to flush out some of the variables that underpin the main measurement objective.
The thrust here is to get behind and underneath the subject, to decompose it so that its constituent parts or contributory factors can be exposed. Often, these turn out to be measurable, although, even then, the challenge of doing so may seem daunting. To encourage us, Hubbard provides four useful insights:
- First, you’re probably not the first person to attempt any particular type of measurement. Research may throw up some useful examples of how others have previously done it. Counting fish in a lake for example.
- Second, you probably already have more information than you know. Most of us don’t realize how much data we already track in our own organizations. So just ask.
- Third, you may not need as much data as you imagine to reduce the uncertainty behind a particular decision. As Hubbard puts it, “the first few observations are usually the highest payback in uncertainty reduction for a given amount of work.”
- And finally, additional information is likely more accessible than you realize, by just thinking outside the box about possible measures. An orchestra, for instance, was able to measure its improvement in performance merely by counting the increase in the number of standing ovations it received.
In fact, what the orchestra was doing in this example was responding to the uncertainty of whether or not its performance was improving and seeking a measurement to reduce that uncertainty. Uncertainty can also be measured as a probability – for example, a 60% chance that a market will grow, a 30% chance it will stay as it is or a 10% chance it will shrink.
Going one step further, measurement specialists talk about levels of confidence attached to ranges of probabilities. Most commonly in measurement, they use what is called a confidence interval
(CI) based on a 90% probability that a measurement will fall within a specified range. For example, if you had a 90% CI for the average weight of a jelly bean as being between 0.25 grams and 5 grams, this would mean you were 90% confident that the average weight would fall within that range.
You might be able to see where this is leading us. If you were able to narrow that range, say to between 0.5 grams and 2 grams, perhaps by learning that 1gram is the weight of something else you can easily visualize, like a thimble-full of water, or by weighing a few of the jelly beans, you would have improved your level of certainty. Effectively, you can make a better bet.
Actually, it turns out that most of us are not very good at establishing our 90% confidence level. We tend to be overconfident, producing too narrow a range at first. Hubbard argues though that getting good at it is a teachable skill – first by testing individuals so they identify their overconfidence and then gradually weaning them away from this optimism with repeated tests and refinements. The process is called calibrating, and having calibrated individuals in your organization, that is, people who are good at assessing uncertainty, can make a significant contribution to the measurement of probabilities in your decision-making processes.
The book includes an exercise to enable readers to assess their level of calibration. It’s a quiz in which you have to express your 90% CI on 10 questions (for example, giving a probability range of dates for the year in which Isaac Newton published his Universal Laws of Gravitation) and 10 true or false questions in which you have to circle your confidence level (ranging between 50% and 100%) on your answer. You can test whether you’re expressing your true confidence level by asking yourself whether you would stick to it if you knew you’d win $1,000 if the true answer was within your range. You can find the full test online by doing a Google search for “Hubbard calibration exercise.”
When uncertainty about a decision includes the possibility it will result in a loss or other negative consequence, measurement experts label this uncertainty as “risk.” Again, many of us find this difficult to quantify, preferring to talk about “low,” “medium” or “high” risk, but these terms, Hubbard suggests, are meaningless. For instance, he asks, is a 5% chance of losing more than $5 billion a low, medium or high risk? What about a 5% chance of losing $100?
There’s no need to fall back on such vagueness when there are statistical methods that enable us to quantify and analyze risk. Foremost among these is the so-called Monte Carlo method, which uses inbuilt spreadsheet functions to create a specific value for the level of risk. And, since the technique also tells us about probabilities and thus is central to Hubbard’s claim of being able to measure anything, it’s worth spending a little time exploring and understanding it.
We won’t bother with history but the name of the Monte Carlo risk analysis method is obviously derived from the world of gambling, a paragon of uncertainty and risk. The basic principle of the technique is that when you have a decision to make, there undoubtedly will be several variables that affect the outcome of that decision.
You develop a range of probabilities for each of these variables, then use a spreadsheet function to choose, at random, a number from each of these ranges, to predict an overall outcome. This operation is repeated thousands of times – easy enough in the era of the personal computer – until you have a range of possible outcomes, which can then be plotted on a graph. This will show you what proportion of your possible outcomes fall below the break-even cost of the investment, which is a true quantification of the risk. The chart will also show you where the highest concentration of probabilities lies, again helping to inform your decision.
Here’s a quick example. You’re considering leasing a machine that will cost $400,000 a year. As a result, you estimate your savings per manufactured unit will be $10 to $20 on maintenance, $2 to $8 for labor and $3 to $9 for raw materials. You believe it might enable you to produce 15,000 to 35,000 more units per year.
The math is that your annual savings will equal the sum of the per unit savings multiplied by the number of additional units produced. For instance a maintenance saving of $11.25, labor saving of $2.47, raw materials savings of $8.25 and additional production of 20,000 units (all of these numbers being chosen at random) would give you a total income of $439,400, a $39,400 “profit” on the lease cost.
Through thousands of random calculations, selecting different values for each variable each time – in effect, “what if” situations – your spreadsheet will produce a table and graph showing the range of possible additional incomes that will be generated. In this case, 14% of the scenarios fall below the $400,000 cost of the machinery. That’s your risk of making a loss.
(You can actually download this spreadsheet example from the book’s website –www.howtomeasureanything.com
Interestingly, for such a straightforward method of quantifying risk, Monte Carlo analysis is not in widespread use. “In fact,” says Hubbard, “the lack of more widespread use of Monte Carlo simulations may be causing organizations to give up major benefits and expose themselves to significant and avoidable risks.”
Of course, this result still doesn’t tell you if 14% is an acceptable risk or not. This is down to the levels of risk tolerance in your business, something which needs to be actively discussed and, if possible, documented, so outcomes such as those generated by Monte Carlo analyses can be weighed against them.
Hubbard, and everyone else in the business, refers to a measurement method as an instrument, broadening the analogy of any piece of equipment used to observe features and elements that otherwise can’t be seen without them, such as a telescope or a set of scales. The instrument makes the measurement manifest. Instruments usually also are consistent, they can be calibrated to take account of errors, they record results and, usually, they are faster and cheaper than unaided human effort. These, too, are features we should expect to see in the methods we use in any measurement quest.
An effective measurement process follows certain key steps:
First, as mentioned earlier, we have to decompose the subject into as many constituent parts as we can possibly identify. In the process, we may discover that there are many elements that do not need to be measured as they don’t affect the outcome. Hubbard even suggests the decomposition may itself remove sufficient uncertainty so no further observations are needed.
Second, assuming we’re not the first to try to measure this topic, we research it, using such sources as Internet search engines, online encyclopedias, industry magazines, academic journals, and government websites. Here’s a tip – use search terms that include words like “survey,” “correlation” and “table”; they’re more likely to take you in the right direction.
Next, consider the observability of the aspects you are trying to measure. What would you see if you were trying to detect them? Do they leave an identifying trail of any kind? For instance, out-of-state shoppers might be identified by license plates in a store’s parking lot, or online gift buyers could be identified by allowing them to add a free gift message to their purchase (as Amazon does).
Once you know what you might be looking for (and surely this is a subject worth brainstorming within your business), establish just how much effort and money will need to be invested into taking the measurement. You do this, as mentioned earlier, by quantifying the value of the information you will gain from doing so. Calculating what Hubbard refers to as the expected value of information, or EVI, relies on another statistical calculation – the expected opportunity loss, or EOL, which is simply the cost of making the wrong decision.
As a simple example, consider an advertising campaign which, if successful, could make up to $40 million. Your calibrated experts say it has only a 60% chance of success. If you didn’t get the go-ahead, the EOL would be calculated as 60% of $40 million, or $24 million. If you could change those 60% odds in your favor by learning something new, thereby reducing uncertainty, the difference between the old and the new EOL would logically represent the value of that information.
So, imagine that you could increase the probability of success by gaining new information, perhaps through a survey, to 70%. Your new EOL would be 70% of $40 million, which is $28 million. So, you’ve improved your EOL, which previously was $24 million by $4 million. That’s your EVI, the value of the information you’d gain by doing the research. As a rule of thumb, Hubbard recommends spending between 2% and 10% of your EVI in acquiring the information.
The instruments you then use to conduct your measurement might include direct observation, surveys and sampling or experiments and statistical modeling techniques that include regression analysis and correlation methodologies, both of which involve measurement of the relationship and dependencies between variables . The important point, says the author, is that you don’t need great precision if all you are seeking is more certainty, which may mean that you don’t need huge studies. Better to start small and be iterative in building on them – you can learn an awful lot from a very small study.
At the same time, you should give some thought to possible biases and errors that might creep into your observations. The most common forms of bias are expectancy (put simply, seeing what you want to see), observer bias in which the very fact of observing influences what is actually happening (which is why scientists use placebo control groups when trialing new drugs), and selection bias in which the “population” you choose to study is not chosen at random.
Finally, in this section, Hubbard devotes a complete chapter to one of the most powerful measurement tools at our disposal, Bayesian statistics. This method is based on the fact that with most measurements, we already have some prior knowledge of the subject matter, which we update as we acquire new information. At its simplest, Bayesian measurement relies on pure instinct – we start with calibrated estimates for our measurement based on what we know, which gives us a degree of confidence in a particular probability. We gather additional information then recalibrate our estimates without doing any additional research, increasing our confidence in the probability.
An illustration would be the jelly bean example mentioned earlier. We have a certain degree of confidence of the likely weight range of the jelly bean from earlier experiments or historic data about other candies, but acquiring the comparative information about the weight of a thimble-full of water and perhaps some sample weights, enables us to increase our confidence by reducing the probable weight range.
More advanced Bayesian inference, using mathematical formulae and algebraic maneuvers enables us to conduct much more sophisticated calculations for all sorts of intangibles, including, for instance, the comparative effect on brand reputation of a number of internal catastrophes in a business, or the probability that a new product launch will be a success if it was also a success in trial marketing.
Humans, Technology and Technique
In the final section of the book, Hubbard takes us beyond the basics to explore issues of human fallibility, newly emerging measurement instruments, and an introduction to his own Applied Information Economics.
The trouble with being human, he points out, is our unreliability and inconsistency of behavior. As survey subjects, for instance, we don’t always say what we believe and our actions often differ from our declared preferences. Not surprisingly, observations of behaviors are generally more reliable indicators than answers to questions. But when we do have to ask questions, especially with regard to intangibles, a useful way of getting people to quantify their responses is by asking them their willingness to pay for something – for example a safer car, improvements in the environment or saving endangered species.
Respondents are also influenced by other factors. For instance, they may be affected by how others respond to a particular question. There’s a famous textbook example of this in which students were asked to say which of three lines was the same length as a fourth line. Although this was visually quite clear, participants began to change their responses when others, who had been planted in the audience and primed to do so, gave the wrong answer.
Respondents also sometimes give answers that they think confirm an earlier opinion they expressed. In one experiment, shoppers were asked to state which of two jams’ tastes they preferred. After doing this, the jars were switched around while they weren’t looking. Asked again to taste and say which they preferred, three-quarters of them selected the jar in the position of their original favorite.
But while untutored and uncalibrated humans can be an impediment to measurement accuracy, 21st century technology is providing new instruments that can significantly enhance our ability to measure, especially when combined with each other. For instance, using GPS, wireless networks, and the freely-available mapping utility Google Earth, one company is able to analyze trucker driving habits and stop times. And another firm uses electronic ID tags at conferences to measure which delegates interact with each other.
Naturally, the Internet has become a key source of measurement data and its potential seems to be increasing exponentially. As an example, a Canadian researcher was able to track flu outbreaks by identifying the locations of people doing Google searches on the subject, giving him a one-week lead on traditional infection notifications from health authorities.
Most recently, the Internet has spawned the development of prediction markets where participants can effectively bet, with real or play money, on the outcomes of just about any event or activity, from the likelihood a company will go bust to the names of the next Oscar winners. Probabilities can be calculated from the prices at which “shares” on these markets trade and research shows them to be remarkably accurate. Companies like Dow Chemical and GE are using them to predict the marketability of new products. To sample a prediction market, visit Foresight Exchange at www.ideosphere.com
, where you can participate with play money.
Finally, Hubbard reveals to us his own measurement process, which forms the foundation of his Applied Information Economics. Not surprisingly, we learn that most of this has already been detailed in the earlier chapters of the book. He breaks it down into a number of phases:
- Project preparation, including initial research, the identification of experts who will contribute to the process and the setting up of workshops to explore the subject.
- Identifying the problem to be analyzed and the factors that will matter in making a decision.
- Calculating the value of information on each variable to decide which ones are worth focusing on.
- Agreeing on the measurement techniques, including decomposing, sampling and experiments.
- Performing the measurements and updating your understanding of the problem and the decisions variables.
- Implementation of decisions that flow from the measurements.
The basis of Doug Hubbard’s contention that we can measure anything – including intangibles – is his assertion that measurement is not so much about precise numbers as it is about probabilities that enable us to reduce uncertainty. Because people don’t realize this, he suggests, individuals and companies are probably not measuring what they should be measuring. As a result, opportunities are lost and serious decision errors can be made.
He encourages us to think creatively to devise ways of quantifying the variables that underlie tangibles and intangibles and to use ranges and probabilities rather than specific numbers to express our degrees of confidence in outcomes and the level of risk associated with those outcomes.
Sometimes, even just a small reduction in uncertainty can have a profound effect on our decision-making. Other times, just a small amount of research can result in a significant reduction of uncertainty. Either outcome is a valuable goal worth striving for.]]>