“Don't Worry about the Ethics of Self-Driving Cars”
Cathy O'Neil, Bloomberg View, April 6, 2018
The problem arises from the subtlety of most algorithmic failures. Nobody, especially not the people being assessed, will ever know exactly why they didn't get that job or that credit card. The code is proprietary. It's typically not well understood, even by the people who build it. There's no system of appeal and often no feedback to improve decision-making over time. The failures could be getting worse and we wouldn't know it.
A while ago, journalists were writing about how good Silicon Valley companies are with software and how surprisingly bad they are with hardware such as drones and spaceships. I think that's dead wrong. Not because startups have been building great delivery drones, but because there's absolutely no reason to think they're doing much better with software. We simply don't know how to look for their failures.