How to Cure a Bad Case of Metric Fixation

How to Cure a Bad Case of Metric Fixation

The Tyranny of Metrics

by Jerry Z. Muller, Princeton University Press, 2018

The Tyranny of Metrics is a pearl of a book. And like all pearls, it was born of an irritation. As chair of the history department at the Catholic University of America, Jerry Muller found himself — and his school — devoting more and more precious time and resources to performance measurement and reporting.

Some of this activity was useful, admits Muller. “But much of the information was of no real use, and indeed, was read by no one,” he writes. “Yet once the culture of performance documentation caught on, department heads found themselves in a sort of data arms race.” In one instance, Muller was urged to add more statistical appendixes to a yearlong department survey, because “the report would look less rigorous than that of other departments” without them. Moreover, the university found itself hiring more and more data specialists, up to and including a new vice president for assessment.

This experience, which Muller describes as an irritating pinprick, inspired him to look more closely at metrics. And that investigation resulted in a concise and clearly argued polemic, which, Muller writes, is “for anyone who wants to understand one of the big reasons why so many contemporary organizations function less well than they ought to, diminishing productivity while frustrating those who work in them.”

The problem is not metrics per se, according to Muller. Rather, it is a “metric fixation” that has become embedded throughout the organizational landscape of society. This fixation values numerical indicators over judgment informed by experience and talent; it equates transparency with accountability; and it holds that rewarding people based on their measured performance is the best way to motivate them. “Metric fixation is the persistence of these beliefs despite their unintended negative consequences,” argues Muller.

Muller explores these consequences in the core of the book, with chapters devoted to institutional realms including education, medicine, policing, business, nonprofits, and the military. For instance, in the chapter on business, he calls out pay-for-performance as a driving force behind several recent corporate scandals. “Performance indicators can certainly aid, but not replace, the key functions of management: thinking ahead, judging, and deciding,” Muller writes.

The history professor traces the inherent problems in metric fixation to a set of recurring flaws in how we use metrics. He finds that we naturally tend to measure what is easily measurable; that we tend to focus on only one or a few measures even when outcomes we are trying to measure are complex; that we tend to measure inputs rather than outputs; and that we strip meaning out of measures by trying to standardize them.

The result, says Muller, is not only a distortion of information, but also a redirection of managerial effort. For instance, focusing on what is easily measurable, such as quarterly revenue and profits, deflects attention from what is not, such as motivation, trust, and cooperation. “For all these reasons, reliance upon measurable metrics is conducive to short-termism, a besetting malady of contemporary American corporations,” writes the author.

Muller also calls out a second set of recurring flaws that are rooted in how we respond to metrics — better known as “gaming.” These flaws include creaming (excluding anything that will lower the numbers), lowering standards (to make the numbers seem higher); distortions or omitting data; and outright cheating.

In 2009, for example, when Medicare began publicly reporting hospitals with higher-than-average 30-day readmission rates, and later began financially penalizing them for the same thing, hospitals began gaming the system. They hid readmissions by billing many of them as outpatient services. “That meant that about half the drop in readmissions was actually due to patients who had in fact returned to the hospital but were treated as outpatients,” reports Muller.

Although Muller indicts metric fixation in The Tyranny of Metrics, he doesn’t suggest that we abandon metrics altogether. “Ultimately,” he concludes, “the issue is not one of metrics versus judgment, but metrics as informing judgment, which includes knowing how much weight to give to metrics, recognizing their characteristic distortions, and appreciating what can’t be measured.” Muller contributes to this effort by including a 10-point checklist for successful performance measurement that in itself is worth the cover price of the book.

Published at Mon, 23 Apr 2018 05:00:00 +0000