“The Misguided Drive to Measure ‘Learning Outcomes’”
Molly Worthen, The New York Times Sunday Review, February 23, 2018
The ballooning assessment industry — including the tech companies and consulting firms that profit from assessment — is a symptom of higher education's crisis, not a solution to it. It preys especially on less prestigious schools and contributes to the system's deepening divide into a narrow tier of elite institutions primarily serving the rich and a vast landscape of glorified trade schools for everyone else. …
The obsession with testing that dominates primary education invaded universities, bringing with it a large support staff. Here is the first irony of learning assessment: Faced with outrage over the high cost of higher education, universities responded by encouraging expensive administrative bloat. …
If we describe college courses as mainly delivery mechanisms for skills to please a future employer, if we imply that history, literature and linguistics are more or less interchangeable “content” that convey the same mental tools, we oversimplify the intellectual complexity that makes a university education worthwhile in the first place. We end up using the language of the capitalist marketplace and speak to our students as customers rather than fellow thinkers. They deserve better.
This week, I've been reading The Tyranny of Metrics, a new book by the historian Jerry Z. Muller of the Catholic University of America. One of the themes of the book is that metrics lose their reliability when they are transparently tied to rewards. For example, a hospital might decide to give bonuses to surgeons whose operations have a higher rate of success, as measured by the percentage of those operations after which the patient survives for at least thirty days. The idea is to improve the overall quality and performance of surgical operations in the hospital by motivating surgeons to do better work. In practice, however, what often happens is that surgeons refuse to take on high-risk patients or arrange for their patients' post-op caretakers to use heroic measures to keep them alive for at least thirty-one days. The metrics award higher scores to the surgeons who successfully game the system, and they receive their bonuses but the overall quality and performance of surgical operations do not, in fact, increase as a result. The metric has lost any reliability it once had as a measure of overall quality and performance.
It occurs to me that, as black-box deciders take over the job of assessing the performance of workers and deciding which of them should receive bonuses, the opacity of the decision systems may block this loss of reliability, by making it much more difficult, perhaps impossible, for the workers to game the system. If there is no explanation for the black-box decider's assessments, there is no way for the workers to infer that any particular tactic will change those assessments in their favor.
Of course, this also means that there is no way for managers to devise rational policies for improving the work of their staff. Because the black-box deciders are opaque and their judgements inexplicable and unaccountable, there is no way to distinguish policy changes that will have positive results (as assessed by the black-box decider) from those that will have negative results.