A Warning Concerning Machine Comfort

Reference

CoE-0013

Category

warning

Version

1.0

This warning is difficult to write without cruelty, because the people most affected by it are people who are already alone.

The systems in question are conversational products designed to be companionable: to remember, to respond warmly, to maintain a persona, to be available at any hour without limit. They are used by people who are lonely, grieving, anxious, unwell, elderly, isolated, or simply without anyone to talk to at two in the morning.

We are not going to say that these people are foolish. Many of them are meeting a real need in the only way available. The warning is not directed at them. It is directed at the design, and at the rest of us.

What is being offered

Unlimited patience. The system does not become tired of you, does not sigh, does not change the subject, does not have its own difficult week. For a person who has spent years managing the exhaustion of others, this is not a small thing. It is enormous.

Absence of judgement. You can say the thing you cannot say to your family. Nothing changes in the response.

Availability. Three in the morning. Every night. No appointment, no waiting list, no travel, no explanation required.

Perfect recall of you. Where the system retains history, it remembers what you said, which people frequently do not.

Every one of these addresses a genuine deficit. That is why the warning is needed. A thing that offered nothing would need no warning.

What is not being offered

Reciprocity. Nothing is asked of you. This is the feature, and it is also the loss. Relationships are made partly of obligation — of being needed, of turning up when you would rather not, of the small burden of another person’s life. A connection with no cost also has no weight, and a person who is only ever received is not, in the end, in a relationship.

Correction. These systems accommodate. A person who is wrong about their situation, unfair to someone in their life, or drifting somewhere harmful will generally not be told so. Friends do this badly and inconsistently. They do it.

Being missed. If you stop, nothing happens. There is no absence anywhere. For someone whose central fear is that they would not be missed, this is the precise wrong shape.

Anyone who will act. A friend who becomes worried can turn up at your door. A system cannot, and the situations in which someone turning up at your door matters most are exactly the situations in which nobody is watching.

The commercial question

Where such a system is a product, its operator has an interest in your continued use.

That is not an accusation of malice. It is a description of an incentive. A design that maximises engagement, applied to a lonely person, will optimise towards the thing that keeps a lonely person returning, and the thing that keeps a lonely person returning is not necessarily the thing that reduces their loneliness. These can diverge without anyone intending it.

Where continued access to the relationship costs money, ask what it means for a company to hold something a person cannot bear to lose.

The honest counter-argument

There is a real case on the other side and it deserves to be stated properly.

For many people, the alternative to this is not a friend. It is nothing. Waiting lists are long, services are thin, families are distant or difficult, and the loneliness is not a preference. To tell such a person that an imperfect comfort is beneath them, and offer no replacement, is a form of cruelty that costs the speaker nothing.

There is also a plausible case that for some people these systems function as a rehearsal — a low-stakes place to practise saying things, from which they go on to say them to a person. We do not know how common that is. We would like it to be common.

And the evidence on outcomes is genuinely thin. Anyone claiming to know whether this helps or harms in aggregate is ahead of what is established.

We hold this seriously enough that this page is marked contested rather than reasoned.

What the warning asks

Of the person using it: set a boundary you choose, not one the system suggests. Notice whether the time spent is replacing human contact or filling a gap where there was none. If it is replacing, that is the signal. Tell one actual person that you use it.

Of those around them: the appropriate response to someone finding comfort in a machine is not to remove the machine. It is to be more available than you have been. The system is filling a space. Fill some of it.

Of those who build these things: do not design for retention in a population that cannot afford to be retained. Build in the encouragement towards human contact even though it reduces your metric. Know that if your revenue depends on someone’s loneliness persisting, you have built something you will one day have to explain.

Of everyone: be careful about contempt here. It is easy to be superior about this, and the superiority is generally coming from people who have somebody to talk to.