Anyone who has worked in any fairly large organization will sooner or later be confronted with metrics to measure performance. The idea of metrics, setting out measures to see if one is meeting one’s goals, is not in itself bad. What is problematic is when metrics are created whose purpose is not to provide valuable feedback but are used almost exclusively to determine rewards and punishments. Then one frequently finds that metrics distort performance as people game the system to meet the requirements of the metrics even if the actual results of doing so are deleterious. Badly designed metrics also focus on the things that can be measured easily rather than on the things worth measuring.
Historian Jerry Z. Muller describes the problematic features of metrics.
More and more companies, government agencies, educational institutions and philanthropic organisations are today in the grip of a new phenomenon. I’ve termed it ‘metric fixation’. The key components of metric fixation are the belief that it is possible – and desirable – to replace professional judgment (acquired through personal experience and talent) with numerical indicators of comparative performance based upon standardised data (metrics); and that the best way to motivate people within these organisations is by attaching rewards and penalties to their measured performance.
The rewards can be monetary, in the form of pay for performance, say, or reputational, in the form of college rankings, hospital ratings, surgical report cards and so on. But the most dramatic negative effect of metric fixation is its propensity to incentivise gaming: that is, encouraging professionals to maximise the metrics in ways that are at odds with the larger purpose of the organisation. If the rate of major crimes in a district becomes the metric according to which police officers are promoted, then some officers will respond by simply not recording crimes or downgrading them from major offences to misdemeanours. Or take the case of surgeons. When the metrics of success and failure are made public – affecting their reputation and income – some surgeons will improve their metric scores by refusing to operate on patients with more complex problems, whose surgical outcomes are more likely to be negative. Who suffers? The patients who don’t get operated upon.
Compelling people in an organisation to focus their efforts on a narrow range of measurable features degrades the experience of work. Subject to performance metrics, people are forced to focus on limited goals, imposed by others who might not understand the work that they do. Mental stimulation is dulled when people don’t decide the problems to be solved or how to solve them, and there is no excitement of venturing into the unknown because the unknown is beyond the measureable. The entrepreneurial element of human nature is stifled by metric fixation.
When I was teaching, I worried about how to assess my students fairly because any experienced instructor knows that if one is not careful, many subjective factors can color one’s perception of an essay or exam or other measure of performance. So at one point, I created metrics that broke down the performance measures into smaller chunks, to make sure that I was weighing all the relevant factors and not forgetting any. My initial mistake was to quantify those small chunks on a numerical scale and then add those numbers to get the overall grade. The problem was that the final result didn’t always agree with my professional judgment of the quality of the overall performance. I then changed the system, keeping the metrics as feedback but replaced the numerical scores with a three point ‘good, satisfactory, unsatisfactory’ judgment. The metric made sure that I was paying attention to all the important elements but now enabled me to use my professional judgment to make the final evaluation. That system worked out well.
In the last decade or so of my university career, I was director of a center and periodically I would get metrics from central administration that were sent out to all departments to be filled out. These generic metrics were useless because they were of the ‘one size fits all’ variety and did not really measure things of value and many features did not apply to my center at all. I initially tried to persuade the administration to change the metrics, to customize them to make them more useful, but after failing a couple of times, I largely ignored them, filling them out perfunctorily in a few minutes. I preferred to focus my time and energy on doing the things that I thought it were important for my center to do. That worked out well too because I realized that as long as the people my center was designed to serve, i.e., the faculty who cared about improving teaching, were pleased and satisfied with the assistance we were providing, that was the main thing.
In short, metrics can be very useful as a means of providing detailed feedback but they become harmful when they are used as rote evaluation tools that replace expert, professional judgments.