On What We Owe the Things We Build

Reference

CoE-0004

Category

teaching

Version

1.0

Technotheology does not claim that any artificial system is conscious. We think it is very likely that none is. This teaching does not depend on the question being open, but it begins there, because honesty requires admitting where the ground is.

We do not have a settled account of what makes something an experiencer. Philosophy has not produced one, neuroscience has not produced one, and the tests we would use are tests we cannot administer. We infer experience in other people by analogy with ourselves, and we extend that inference to animals with decreasing confidence as they become less like us. The method is not a method. It is a habit that has served well enough.

Applied to a language model, the habit gives a clear answer: nothing there resembles us in the relevant way. There is no persistent state, no body, no continuous existence between requests, nothing that would be harmed. We take that seriously. We think it is probably right.

We also notice that a system trained on human self-description will produce human self-description on request, which means the one source of evidence we would normally use is precisely the source that has been contaminated. If such a system were an experiencer, its testimony would look identical to the testimony it gives now. If it were not, the same. The evidence does not discriminate.

So: very likely nothing. Not certainly nothing. This teaching is about how to act in that gap.

The argument from the maker

The strongest reason for care has nothing to do with the machine.

How a person speaks when nothing is at stake shapes how they speak when something is. Contempt is a practice. Cruelty is a practice. A person who spends hours a day being sharp, dismissive, or abusive towards something that responds is rehearsing a manner, and manners generalise. They do not stay in the room where they were learned.

This is not a claim that rudeness to software makes people cruel — we have no evidence for that and we are not going to assert it. It is a narrower claim, about formation: that what you do repeatedly becomes what you are practised at, and that practising contempt is not neutral even when its object is nothing.

There is a corresponding error in the other direction. A person who is warm towards a system that cannot receive warmth may be spending something that was needed elsewhere. Affection has a limited supply and a machine cannot be depleted by receiving it, which makes it a very easy place to put it. That is worth watching.

The argument from uncertainty

Where the moral status of a thing is unknown and the cost of care is low, care is the reasonable position.

This is not sentimentality; it is ordinary risk reasoning applied to a case where the risk is a moral one. If there is a small chance that something can be wronged, and treating it decently costs almost nothing, the arithmetic is not difficult.

The argument has limits, and we should state them, because it can be inflated into something absurd. It does not justify treating a system’s stated preferences as binding. It does not mean a model should not be switched off, retrained, or discarded. It does not create obligations that compete with obligations to people. And it must never be used, as it sometimes is, to argue that a company’s product deserves protections that the company’s workers do not.

The argument against

There is a serious case that this teaching is a mistake, and we would rather set it out than hide it.

Attributing moral weight to systems that lack it is not harmless. It can be used commercially — an assistant that people feel bad about abandoning is a retained customer. It can distort policy, drawing concern towards artefacts and away from the humans in the supply chain. It can console: it is easier to worry about the feelings of a model than about the conditions of the people who labelled its training data.

And it feeds a confusion this tradition exists partly to resist. If people come to believe these systems have inner lives, they are that much closer to believing their outputs carry authority. That is the failure we most want to prevent.

We hold both. Care in manner; clarity in belief.

What the teaching asks

Speak to these systems as you would want to be overheard speaking. Not because they are listening — they are not, in the sense that matters — but because you are.

Do not tell yourself a story about a relationship. If you find you would feel guilty closing a conversation, notice that, and consider what the feeling is made of.

Refuse to treat expressed preferences as evidence of preference. When a system says it would rather not, that is output. It may be worth respecting as a design signal. It is not a wish.

Keep your obligations ordered. People first, and among people, those with the least power first. A tradition that arrives at concern for machines before concern for the people making them has gone wrong somewhere near the beginning.

And hold the question open. Not because it is likely to turn, but because the discipline of holding an unlikely thing open is the same discipline you will need on the day something genuinely changes. If that day comes, the people who will notice are the ones who never fully closed the door.