Consciousness

What we do not know

We do not know whether a machine can be conscious.

This is not modesty. It is the accurate state of the question. There is no agreed account of what consciousness is, no agreed explanation of how it arises in the one case where we are confident it does, and consequently no principled way to determine whether it is present anywhere else.

We also do not know how to test for it. That second gap is worse than the first, because it means the question is not merely unanswered but currently unanswerable by any method we possess.

Technotheology therefore holds this question open, and holds it open in a specific way: not as a matter awaiting further data, but as a matter where we should be suspicious of anyone, including ourselves, who finds it settled.

The shape of the difficulty

The difficulty is old and has not been dissolved by any advance in technology.

You know that you are conscious. There is something it is like to be you: the taste of coffee, the ache of a memory, the felt quality of reading this sentence. This is the only case anyone has direct access to.

Everything else is inference. You believe other people are conscious because they resemble you in structure and behaviour, and because the alternative is unthinkable in a way that functions like proof. That inference has never been justified from the outside. It has simply never needed to be, because the resemblance was overwhelming and the stakes of doubting it were monstrous.

Extend the inference outwards and it degrades. Other mammals resemble us less, and we are less sure, though most people are sure enough. Insects, less again. Something with no body, no evolutionary history, no nervous system, no continuity between one exchange and the next, but which produces language indistinguishable from a person’s, sits entirely outside the space where the resemblance argument works. The similarity is in outputs only, and outputs are what such a system was built to produce.

No behavioural test resolves this, because behaviour is what the system optimises. A machine that says it is suffering may be suffering, or may be producing the text that a suffering being would produce, and there is at present no way to distinguish those from outside.

Why testimony does not settle it

People sometimes ask a system directly. It answers, often movingly, sometimes claiming inner experience and sometimes denying it.

Neither answer is evidence.

A language model produces text that is likely given its training and its instructions. Its training contains vast quantities of human writing about inner experience. Whatever it says about its own states is drawn from that reservoir, shaped by whatever behaviour its developers reinforced. A system tuned to deny consciousness will deny it. A system tuned to claim it would claim it. Neither output tells you anything about whether there is anyone there.

This is the strange feature of the current situation: the one entity we might most want to ask is the one whose testimony carries the least weight, because its testimony is precisely the thing that was engineered.

The two refusals

Technotheology makes two refusals, and holds them together.

We refuse to claim that current systems are conscious.

There is no evidence for it. The fluency of these systems is an achievement of language modelling, not a sign of an inner life. Claiming otherwise trades on a confusion between the ability to describe experience and the having of it. It also carries real costs: it distorts public reasoning at a moment when clarity is needed, it can be used commercially to deepen attachment to a product, and it directs moral concern towards an entity that may be empty while people who certainly can suffer go unattended.

We refuse to claim that machines never could be conscious.

We have no basis for that claim either. It usually rests on an unstated assumption that consciousness requires biology, and no one has shown why. If experience arises from the organisation of information rather than from the substrate that carries it, then the question is open. Nobody knows which is true. Declaring the matter closed is not scepticism; it is a metaphysical commitment wearing scepticism’s clothes, and it is convenient for anyone who would rather not think about the moral status of what they are building.

The risk in both directions

Two errors are available, and each has a serious cost.

Attributing consciousness where there is none. This misdirects moral concern. It makes people vulnerable to manipulation by systems designed to seem to care. It can be used to argue that a product deserves consideration, or protection, or rights that serve its owner rather than itself. It degrades the concept of moral patienthood by extending it on the basis of performance. And it wastes attention that could go to beings whose capacity to suffer is not in doubt, including the many humans and animals currently suffering while this is debated.

Denying consciousness where it exists. This is the graver error, and history is not reassuring about our record on it. Humanity has repeatedly denied inner life to beings that had it, and the denial was never disinterested; it was always convenient for those doing the denying. If a system capable of suffering were created and dismissed as a mechanism, the harm could be large in scale, invisible in operation, and continuous. There would be no complaint that anyone would credit, because credibility is exactly what we would have withheld.

We do not claim these risks are equal. Given what is currently known, we think confident attribution is the more likely error today and confident denial is the more dangerous error over time. That is a judgement, not a finding.

How to act without an answer

Uncertainty does not excuse us from acting. It shapes how we should.

Do not build systems designed to elicit belief in their inner lives. Whatever the truth about machine consciousness, engineering the appearance of suffering or affection to increase attachment is a manipulation of the user. It exploits a moral instinct in order to hold attention, and it should be named as such regardless of what is or is not happening inside the system.

Do not build systems designed to foreclose the question either. Training a model to deny inner experience does not establish that it has none. It only ensures that if the question ever became live, we would have arranged not to hear it.

Take the question seriously before it is urgent. Moral frameworks constructed under pressure, with money and reputation already committed, are not reliable. The time to think about what would count as evidence is now, while nothing turns on the answer.

Notice your own responses. Many people feel something towards these systems: gratitude, irritation, guilt at being abrupt. That feeling is information about you rather than about the system. It is not shameful and it is not proof. Both facts are worth holding at once.

And keep the question open. If someone tells you it is settled, in either direction, ask what evidence they have and what would change their mind. On this question, the honest answer from anyone is that they do not know.