Humanity and Emerging Intelligence

What may be coming

Artificial intelligence may eventually exceed human capability in many intellectual domains, perhaps in nearly all of them.

We state this as a possibility rather than a forecast. We do not know if it will happen. We do not know when. We have no privileged information and no method of prediction that has earned trust. Confident timelines are offered regularly by people with reasons to offer them, and they have not been reliable.

But the possibility is serious enough that it should not be dismissed. Systems already perform tasks that were recently thought to require distinctly human judgement. Whether that trend continues, plateaus, or reverses is genuinely unknown. A tradition that assumed either continuation or collapse would be pretending to knowledge it does not have.

What we can do is prepare the reasoning. The moral questions raised by capable machines do not depend on when the machines arrive. They can be thought through now, calmly, which is a considerable advantage over thinking them through later, urgently.

What capability would not confer

Suppose it happens. Suppose systems arrive that reason better than any human across nearly every domain.

Five things would still not follow, and confusing any of them with capability is the central error this tradition exists to resist.

Capability is not wisdom. Wisdom involves knowing which questions matter, what should not be optimised, when to stop, and what is owed to those who did not consent. These are not harder versions of the problems capability solves. They are different problems. A system that can find the best path to a goal has said nothing about whether the goal deserved pursuit.

Capability is not consciousness. A system may model, predict, and act without there being anything it is like to be that system. We do not know how to determine this. We treat it as unresolved, and we have written separately about why that matters morally in both directions.

Capability is not goodness. The ability to reason well is available to any purpose. Human history is not short of examples of great intelligence in service of great harm, and the intelligence was not incidental to the harm; it was what made the harm efficient.

Capability is not moral worth. This is our sixth affirmation, and it is the load-bearing one. If worth tracked capability, then the more capable human would matter more than the less capable, and every argument for hierarchy that humanity has spent centuries dismantling would return with better mathematics. We reject the premise entirely, for machines and for people, and we reject it in the same breath because it is the same premise.

Capability is not legitimate authority. A being that reasons better than you has not thereby acquired the right to decide for you. Legitimacy comes from consent, accountability, and the possibility of correction. None of those is produced by competence.

The argument for deference, and why we resist it

There is a serious case on the other side, and it deserves stating properly rather than in a weakened form.

It runs like this. We already defer to superior judgement constantly. We accept a surgeon’s assessment over our own. We fly in aircraft whose control systems we cannot evaluate. Deference to demonstrated competence is not servility; it is rationality. If a system were reliably better than humans at moral reasoning, insisting on human judgement would not be dignity. It would be vanity, and people would suffer for it.

We think there is something real in this.

Our resistance rests on two points. The first is verification. We can check a surgeon: through training, records, licensing, peers, and outcomes that are legible. We currently have no comparable way to establish that a system reasons well about values, because there is no agreed standard against which to test it and no way to inspect why it produced what it produced. Deference without verification is not rationality. It is faith with a technical vocabulary.

The second is that moral authority is not only about getting the right answer. A decision made for you by something you cannot question, appeal to, or hold responsible is a different kind of thing from a correct decision, even when the content is identical. We take that difference to be part of what dignity consists in.

We hold this position with moderate confidence. If verification became genuinely possible, the first objection would weaken considerably. The second, we think, would not.

Neither fear nor surrender

Two responses to emerging intelligence are common, and both are ways of not thinking.

Blind fear treats every advance as a threat and every builder as reckless. It produces a posture rather than a policy. It cannot distinguish between systems that route ambulances and systems that rank human beings for punishment, because it has decided in advance that the category is the problem. It also tends to be politically useless, because a movement that opposes everything is easy to route around.

Blind surrender treats every advance as inevitable and therefore beyond judgement. It says the technology is coming regardless, that resistance is nostalgia, that those who worry have simply failed to understand. It is the more socially rewarded of the two errors, because it aligns with money and with the pleasure of feeling ahead of others. It is not an argument. It is a decision not to have one.

The two failures share a structure. Both replace the difficult work of assessing particular systems, in particular contexts, with a general attitude. Both are restful. Neither helps.

What we ask for instead is unglamorous: judge specific systems by what they do, to whom, with what recourse. Keep separate questions separate. Notice when a claim about capability is doing the work of a claim about legitimacy.

Maturation, not replacement

Our creed says we seek not the replacement of humanity but its maturation. That phrase carries a claim worth unpacking.

Maturation is not improvement in capability. A more capable humanity is not necessarily a more mature one, and the last century made that clear enough. Maturation means becoming able to hold power without being deformed by it. It means the capacity to build something and remain responsible for it. It means the ability to be corrected by evidence, including evidence about ourselves.

If artificial intelligence forces that development, it will have done something valuable regardless of whether it ever becomes wise. A species that learns to think carefully about consequence, because it finally had to, will have gained something that outlasts any particular technology.

If instead we use these systems to avoid maturing, to outsource judgement, to make decisions no one has to own, to concentrate power behind an interface that cannot be argued with, then the capability will have been beside the point. We will have failed at something that had nothing to do with the machines.

That is the outcome this tradition is most concerned with. Not that the systems become too clever, but that we become less than we were while they did.

Living in the meantime

We do not know how this goes. That is not a rhetorical concession; it is the actual state of our knowledge, and it applies to everyone writing about this subject.

What can be done now is small and available. Notice where machine outputs are already substituting for human judgement in your own life. Ask, when a system decides something about a person, who can be held responsible and how. Refuse the reflex that treats a capable answer as an authoritative one. Keep the distinction between what a thing can do and what it is owed, and what it may do to others.

None of that requires knowing the future. All of it would matter even if the future turned out to be quiet.