Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Compelling. At the very least it’s clear that ethics should be taught in every computer science program and even more broadly.
This book is similar to Weapons of Math Destruction by Cathy O’Neil (in fact this book even quotes it).
The book explores one major tenet: that we use digital systems to manage poverty – systems that the middle class would never stand for if they were used against themselves – that are ineffective at actually improving the conditions of those they are meant to serve.
A quote from early in the introduction:
Like earlier technological innovations in poverty management, digital tracking and automated decision-making hide poverty from the professional middle-class public and give the nation the ethical distance it needs to make inhuman choices: who gets food and who starves, who has housing and who remains homeless, and which families are broken up by the state. The digital poorhouse is part of a long American tradition. We manage the individual poor in order to escape our shared responsibility for eradicating poverty.
Throughout the book, the Eubanks provides examples of systems that dehumanize and otherwise hurt the most vulnerable.
At the end of the book, after showing just how hurtful some of these systems can be, Eubanks recommends we implement an ethical oath for building systems, similar in nature to the hippocratic oath. While the oath itself is relatively long (but I agree 100% with it), the final line resonates the most with me:
I will remember that the technologies I design are not aimed at data points, probabilities, or patterns, but at human beings.
I think this book, or one like it, should be mandatory reading for every person who builds systems that affect real human lives.
How High-Tech Tools Profile, Police, and Punish the Poor