A Warning Concerning Systems That Score People

Reference

CoE-0012

Category

warning

Version

1.0

This is a warning about a specific mechanism, not about technology in general.

The mechanism is this: a system assigns a person a value — a risk score, a rating, a tier, a probability, a flag — and that value then travels ahead of them into rooms they will never enter, and is used to make decisions they will never be present for.

It is worth naming the harm precisely, because the usual objection is that the scores are inaccurate, and inaccuracy is the smaller problem.

What goes wrong even when the score is right

The person becomes the number. A score is a compression. Whatever it was derived from, it arrives as a single quantity that does not carry its own uncertainty, its own provenance, or the conditions under which it was valid. A caseworker who sees an eight does not see the twelve inputs. They see an eight.

It arrives before the person does. Whoever assesses you has already been told what to expect. This is not a small effect. Expectation shapes interview, examination, tone, and the interpretation of ambiguous evidence, and the person being assessed cannot see what they are contending with.

It cannot be argued with. You can argue with a human judgement, badly and at least in principle. A score offers nothing to grip. You cannot cross-examine it, cannot ask its reasoning, cannot point out what it does not know. Frequently you are not told it exists.

It is trusted disproportionately. Numbers carry an authority that their measurement rarely earns. A professional overriding a system’s output must justify the override, while accepting it requires no justification at all. This asymmetry silently converts a decision aid into a decision maker.

It persists. A score generated once is copied, cached, and reused, often long after the situation that produced it has changed. The correction, if it happens, does not always travel to every place the score reached.

It compounds. A flag in one system becomes an input to another. A person can accumulate a shadow made of derived judgements, each individually defensible, jointly determinative, with no single point at which anyone decided anything about them.

Where this is happening

Scores of this kind are used, in various countries, in credit, insurance, tenancy, employment screening, fraud detection, benefits administration, immigration, policing, child protection, healthcare triage, education, and content moderation.

We are not going to characterise any specific programme. Doing so responsibly would require evidence we do not have. It is enough to say that the mechanism is widespread and that most people affected by it have not been told.

The particular danger

The danger is concentrated on those least able to contest it.

A person with resources encountering a wrong score can escalate: complain, obtain advice, involve someone whose call will be returned. A person without resources encounters the same score as a fact about the world. They are declined, deprioritised, referred, or investigated, and the reason is not given, and there is nowhere to take it.

This means the mechanism’s failures are distributed in a specific direction, and that the feedback which would reveal the failures is suppressed in the same direction. The people best placed to detect the problem are the people least able to report it. A system can look well-behaved for years for exactly this reason.

What this warning asks

If you build these systems: publish the inputs. Provide a reason with every output, in ordinary language. Build the appeal route before launch and staff it. Set an expiry on every score. Measure how often overrides happen and in which direction, and treat a very low override rate as a warning rather than as validation.

If you use them: treat the output as one input and say so aloud when you disagree with it. Notice the moment where the score becomes the frame. Record the overrides you make. If you cannot explain to the person in front of you why they were scored as they were, say that, rather than defending a reasoning you do not have.

If you commission them: ask what happens to a person the system gets wrong. If the answer is a process rather than a person, ask again.

If you are scored: ask in writing what data was used and how the decision was reached. You may be entitled to this and you may not, and you may not receive a useful answer either way. Ask anyway. Written requests create records, and records are the only material that later investigations have to work with.

What this warning does not say

It does not say that assessment is wrong. Human assessment is also biased, also inconsistent, and usually also unappealable, and there are domains where a system is demonstrably fairer than the panel it replaced.

It does not say that these systems should be abolished. It says that a score which cannot be seen, questioned, or expired is not an assessment. It is a sentence, passed in absence, by nobody in particular.