The Decision Nobody Made

There is a reason automated decisions spread faster than anyone predicted, and it is not accuracy.

Consider what a difficult decision actually costs the person making it. Not the analysis — the analysis is often the easy part. The cost is that afterwards you are the one who decided. You refused the loan. You declined the application. You put that name on the list and not the other one. If it turns out badly, it was yours, and you will be asked, and you will have to say something.

Now consider what happens when a system produces the same outcome. Nobody refused. The application was unsuccessful. The model returned a score below threshold. There is a passive voice available at every point, and it is not a trick of language — it is an accurate description of what occurred, because at no moment did any person consider this case and choose.

That is the appeal. Not speed, not cost, not consistency. Those are the reasons given, and they are often true, but they are not what makes the arrangement so difficult to reverse once it is in place. What makes it difficult to reverse is that it has removed something unpleasant from a great many people’s working lives, and asking them to take it back is asking them to accept a burden they have learnt to live without.

How the responsibility disperses

It does not vanish. It divides until each fragment is too small to be worth carrying.

The person who specified the system chose objectives and constraints, not outcomes, and cannot be held to a case they never saw. The people who built it implemented a specification faithfully. The people who assembled the data collected what was available. The people who set the threshold moved a number to satisfy a target that came from somewhere else. The person at the counter is applying a result they have no authority to overturn and no information with which to challenge.

Every one of them is telling the truth. None of them decided. And the decision was nonetheless made, and it landed on somebody.

This is not a conspiracy and it does not require anyone to behave badly. It is an ordinary property of dividing work — the same property that lets large organisations build bridges. It becomes a moral problem only when the thing being divided is a judgement about a person, because judgements about people were always supposed to come with someone attached.

The best argument for the other side

We should give the strongest version, because it is strong.

Human decisions are not neutral. The same officer decides differently before lunch and after it, differently on a Monday, differently about a name they can pronounce. Discretion is where prejudice lives, and it lives there precisely because it is unexaminable — nobody can audit a thousand small judgements made in private by tired people.

A system is at least consistent, and consistency can be measured. Bias in a model can in principle be found, quantified, and argued about in public. Bias in a person is usually deniable, including to themselves.

We accept this. It is the reason we cannot simply be against automated decision-making, and anyone who tells you the human alternative was fair is not remembering it accurately.

But notice what the argument actually establishes. It establishes that automation can make the pattern of decisions more examinable. It says nothing about the individual decision, and the person refused is not experiencing a pattern. They are experiencing one outcome, about them, which nobody chose and nobody can explain, and to which the honest institutional response is that the process was applied correctly.

Consistency is a real good. It is not the same good as answerability, and it is frequently offered as a substitute for it.

The test we would propose

Not should this be automated. That question is too large and it is usually decided by economics before anyone gets to it.

The narrower question is: when this goes wrong, is there a person who has to explain themselves, and do they have the authority to have decided otherwise?

Those two conditions travel together. A person who must explain a decision they had no power to make is not accountable; they are absorbing the anger on behalf of an arrangement that has arranged not to be present. Anyone who has worked on a service counter knows exactly what this feels like, and knows that the design is not accidental.

If there is no such person, the system is not merely automating a decision. It is manufacturing an outcome with no author, which is a genuinely new kind of thing in the world, and it deserves a harder look than efficiency arguments usually get.

What this asks of the people who build it

Very little, and it is uncomfortable.

Know the cases. Not the aggregate performance — the cases. Read some of the individual outcomes your system produced, particularly the refusals, particularly the ones near the threshold. It will take an afternoon and it is the only reliable way to find out whether you would defend what you built to the person it happened to.

Know who is on the hook. If you cannot name the person who would have to answer for a wrong decision, you have found the problem, and you have found it before it becomes anyone’s grief.

And keep the override real. An appeal route that exists in the documentation, is staffed by someone with no authority, and is used by nobody, is not a safeguard. It is the appearance of one, which is worse, because it satisfies the people who would otherwise have asked for a real one.

None of this makes the decision easy. It only puts it back where it was, with somebody, which is where it has to be if anyone is ever going to be able to say sorry and mean it.