There is something irresistible about disagreeing with everybody in the room. Not
because everybody is necessarily wrong; quite often they are right. But unanimity
makes me nervous. Whenever a sufficiently large number of intelligent people begin
nodding gravely at one another and using phrases such as “existential threat,”
“unprecedented danger,” “human extinction” and “the future of civilisation,” a small
and possibly defective part of my brain wants to raise its hand and ask an irritating
question: are we sure?
I should confess immediately that, on artificial intelligence, I am by temperament on
the cautious side. The technology is advancing at a pace that is genuinely
extraordinary. We do not fully understand where it is going. There are foreseeable
risks, unforeseeable risks and, most interestingly, Donald Rumsfeld’s famous
category of unknown unknowns. When a technology begins acquiring capabilities
faster than society can develop the institutions, laws, habits and conventions needed
to contain them, caution is not cowardice. It is common sense.
That is what I actually believe. Unfortunately, believing something has never
prevented me from arguing against it. So occasionally I appoint myself counsel for
the defence, take the opposite side simply to see whether the prevailing argument
survives cross-examination, and once I do that, the case against our current state of
AI panic becomes surprisingly interesting. Humanity has been here before—many
times. We simply have a poor memory for previous apocalypses.
Listen to some discussions around artificial intelligence today and the language is
magnificent. The technology is advancing uncontrollably. We may be creating
something whose consequences we cannot anticipate. It could escape our control,
reproduce its own capabilities, be weaponised, destabilise society or alter humanity
permanently. By the time we understand the danger, it may already be too late. All
perfectly reasonable concerns, except that if you changed a few nouns, you could
have been listening to molecular biologists fifty years ago.
In the early days of recombinant-DNA technology, scientists suddenly acquired the
ability to cut genetic material from one organism and insert it into another. Today this
is Biology 101. In the 1970s it bordered on sorcery conducted with pipettes. A
bacterium could suddenly possess genes that nature had never placed inside it. Viral
material could be moved between organisms. Microbes might be engineered to
manufacture substances they had never manufactured before. And bacteria have
one characteristic that makes them particularly unsuitable for nervous people: they
reproduce extremely well without asking anyone’s permission.
So the obvious question arose: what happens if one gets out?

This was not merely newspaper hysteria. Serious scientists were worried—so
worried, in fact, that researchers voluntarily deferred certain experiments while the
hazards were considered. In 1975, the famous Asilomar conference brought together
leading scientists, physicians, lawyers and others to consider how recombinant-DNA
research could continue without creating unacceptable biological risks. The questions
sound uncannily contemporary. How much containment is enough? Can researchers
regulate themselves? Are we building capabilities faster than we understand their
consequences? What if something created inside the laboratory behaves differently
outside it? What if an organism escapes into the sewage system and begins
reproducing, a harmless bacterium acquires pathogenic properties, or a virus is
altered in ways we cannot reverse?
In other words, the familiar quartet was already present: extraordinary promise,
profound uncertainty, potentially irreversible consequences and a public trying to
decide whether the scientists had become too clever for their own good. Today
recombinant-DNA technology is not usually discussed as an extinction event. It is
used to make insulin.
That does not mean the scientists at Asilomar were foolish. Quite the opposite. Their
caution helped create the framework within which the technology could mature. What
history suggests, however, is something subtler: the existence of serious
uncertainty does not mean that the worst imaginable outcome is the most
probable one. This distinction frequently disappears once television studios become
involved.
Human beings have a peculiar relationship with technology. We invent something,
become terrified of it, regulate it, become accustomed to it and eventually complain
when the Wi-Fi version does not work. The more transformative the invention, the
more metaphysical the panic. Electricity was unnatural; railways were dangerous to
the body; telephones would destroy real conversation; television would destroy the
intellect and eyesight; video games would produce psychopaths; and the internet
would destroy privacy, newspapers, attention spans, childhood, democracy and
spelling. Admittedly, the internet has made a respectable attempt at several of these.
Nevertheless, civilisation limps onward.
Biotechnology gave us one of the most revealing examples of the pattern. Once
scientists could alter organisms, another question appeared: can you patent life?
Today this sounds like the sort of sentence that belongs in a first-year course on
biotechnology law. In the late twentieth century, it sounded like a metaphysical
emergency.
Ananda Mohan Chakrabarty, an Indian-born microbiologist working in the United
States, engineered a strain of Pseudomonas carrying multiple plasmids that enabled
it to break down several components of crude oil. The idea was practical enough: an
organism that might help clean oil spills. Then he tried to patent it. The Patent Office
objected. Processes could be patented, machines could be patented, chemicals
could be patented—but a living organism? Surely not.

The dispute eventually reached the United States Supreme Court, which ruled in
1980 in Diamond v. Chakrabarty that the bacterium was patent-eligible because it
was a human-made organism with characteristics markedly different from those
found in nature. And suddenly an extraordinary philosophical door appeared to open.
If a bacterium could be patented, what came next—a plant, an animal, a human
tissue, a gene, a human being?
The popular imagination did what it always does when handed a technological
breakthrough and five minutes of unsupervised free time: it sprinted directly to
dystopia. Scientists, it was suggested, would begin designing organisms and claiming
ownership over them. Corporations would manufacture life. Biology itself would
become intellectual property. Nature would acquire shareholders.
These were not entirely foolish anxieties. Patent law really did struggle for decades
with genes, plant varieties, modified organisms, human tissues and biological
inventions. But the genetically engineered bacterium did not inaugurate the corporate
ownership of the biosphere. Instead, biotechnology became an industry. Yesterday’s
metaphysical horror became tomorrow’s regulatory affairs department.
And then came Dolly.
In 1997 the world met a sheep, not normally an event requiring philosophical
reconstruction of civilisation. But Dolly was different. She had been born the previous
year at the Roslin Institute in Scotland, created by transferring the nucleus of an adult
mammary cell into an egg whose own nucleus had been removed. From that
reprogrammed adult cell came an entire sheep. An adult cell had, in effect, been
persuaded to forget what it had become and begin again.
That was astonishing. Dolly was not the first cloned animal, but she was the first
mammal cloned from an adult somatic cell. The distinction mattered enormously. She
demonstrated that the genetic information inside a specialised adult cell remained
capable, under the right circumstances, of directing the development of an entire
organism.
The biological implications were profound. The newspaper implications were even
more profound. If you could clone Dolly, the obvious question was: when do we clone
Margaret? Suddenly every magazine cover contained some variation of the same
anxiety. Would dictators reproduce themselves? Would bereaved parents clone dead
children? Would billionaires manufacture younger copies of themselves? Would
armies consist of genetically identical soldiers? Would humans become products?
And lurking beneath all these questions was an older discomfort: had reproduction
ceased to be sacred because technicians could interfere with it?
For a while, Dolly was not merely a sheep. She was an omen. Yet something
fascinating happened: we got used to her.
Cloning did not disappear. Quite the opposite. Mammals including cattle, goats,
horses, dogs, cats and others have since been cloned. Valuable breeding animals

can be replicated. Elite livestock are cloned. Polo ponies have been cloned. Camels
have been cloned. Commercial animal cloning exists. A procedure that once seemed
to herald the end of nature gradually migrated into the less dramatic world of
veterinary reproduction. We went from MY GOD, THEY HAVE CLONED A
MAMMAL to That horse cost how much? Clone it.
This is one of humanity’s more endearing characteristics. Yesterday’s violation of the
natural order becomes tomorrow’s specialist service industry.
The early 2000s then gave us another drama: the human-cloning scandal
surrounding the South Korean scientist Hwang Woo-suk. For a brief period Hwang
was an international scientific celebrity. His laboratory claimed extraordinary
advances in cloning human embryos and generating patient-specific embryonic stem-
cell lines through somatic-cell nuclear transfer.
The implications appeared enormous. One could imagine taking a cell from a patient,
producing a genetically matched embryo, deriving stem cells and perhaps one day
growing tissues that would not be rejected by the immune system. Scientific
immortality beckoned, and so did ethical Armageddon. Had humans been cloned?
Were cloned babies next? Had the final boundary fallen?
Then came one of science’s less glamorous but extremely useful inventions:
verification.
The central human stem-cell claims turned out to be fabricated. Investigations
showed that the supposedly revolutionary patient-specific cloned stem-cell lines did
not exist as claimed. Ironically, Hwang’s laboratory really had cloned a dog. Reality,
as usual, proved less tidy than either the enthusiasts or the prophets of doom would
have preferred.
The Hwang episode is worth remembering in the age of artificial intelligence because
it demonstrates something important: societies can become frightened not merely by
technologies that exist, but by technologies they believe exist. We are perfectly
capable of experiencing the social consequences of a technological breakthrough
before the technological breakthrough has actually occurred. That may be one of the
most relevant lessons for AI, because the perception of capability can itself become
capability.
And now biotechnology has carried us into even stranger territory. Scientists and
companies are seriously discussing de-extinction: the woolly mammoth, the dodo, the
thylacine and other species that disappeared from the Earth thousands of years ago.
Dinosaurs remain, for the moment, safely confined to Jurassic Park. Their DNA is
simply far too ancient for anything resembling straightforward cloning. But the
conceptual boundary between extinction and permanence has nevertheless become
less absolute than it once seemed.
Think about the progression. First we feared modifying life, then patenting life, then
copying life, then copying human life, and now we contemplate reconstructing forms

of life that no longer exist. Humanity adapts quickly. Give us twenty-five years and
yesterday’s moral catastrophe becomes a grant proposal.
This pattern is not unique to biotechnology. Rabindranath Tagore understood the
underlying anxiety beautifully in Muktadhara, his 1922 play built around a great
machine that dams a waterfall. The dam becomes more than an engineering object; it
becomes a symbol of power, human technology arresting the natural movement of
water and, through that intervention, controlling other human beings.
Tagore’s deeper objection was not to engineering in any simplistic sense. It was to
technological arrogance—the moment when the ability to do something becomes
confused with the right to do it. That question remains timeless. Can we do it? Should
we do it? Who decides, who benefits and who bears the risk? Artificial intelligence
deserves exactly those questions.
But notice something else. Humanity did not respond to dams by abandoning dams.
We learned, imperfectly and sometimes painfully, to argue about which dams should
be built, where, under what environmental safeguards, with what compensation, for
whose benefit and at what ecological cost. Progress rarely consists of choosing
between worshipping technology and prohibiting it. Civilisation is mostly the untidy
business of learning how not to misuse things we are absolutely determined to use.
Consider the Three Gorges Dam in China. The project generated immense
controversy, much of it entirely legitimate: displacement of communities, ecological
disruption, geological risks and damage to archaeological landscapes. Yet around
gigantic human projects, legitimate concerns often acquire mythological plumage.
The dam became associated in popular discussion with claims that it might affect the
rotation of the Earth or even disturb the planet itself.
Here comes the delightful part. Moving that quantity of water really can affect the
Earth’s rotation, just not in quite the cinematic fashion one might imagine.
Redistributing tens of billions of tonnes of water changes the distribution of the
planet’s mass and therefore, at an almost fantastically tiny scale, its moment of
inertia. The calculated effect on the length of the day is on the order of a few
hundredths of a microsecond.
So humanity did indeed interfere with planetary rotation. The apocalypse lasted
considerably less than the time required to blink.
This seems to me a useful parable for technological anxiety. A frightening claim can
contain a kernel of scientific truth and still produce a completely disproportionate
intuitive reaction, which brings us neatly back to artificial intelligence.
One of the favourite sentences in discussions about AI is: “What happens when
machines become more intelligent than human beings?” It sounds terrifying largely
because of the definite article hidden inside it: intelligence, as though intelligence
were a single measurable fluid. Humans possess 173 units of Intelligence, GPT-27
achieves 174, and humanity immediately becomes obsolete.

But intelligence does not work like that. A cheetah possesses neural circuitry for high-
speed locomotion that I would hesitate to challenge; an eagle processes visual
information at distances where I would be looking for my spectacles; a bat navigates
using echolocation; migratory birds perform feats of orientation that would leave most
humans standing beside the wrong departure gate staring irritably at Google Maps;
dogs inhabit an olfactory universe we barely perceive; and even insects display
remarkable navigation and collective behaviour with nervous systems so small that
we routinely destroy them with folded newspapers.
Yet nobody convenes an emergency session of the United Nations because a
bloodhound is better than a Nobel laureate at locating a concealed sausage.
Superiority in one cognitive domain does not automatically confer sovereignty.
Human beings did not become the dominant species because we were best at every
biological task. We are comically mediocre at many of them: we cannot fly, cannot
breathe underwater, cannot run particularly fast, have unimpressive night vision and
possess a sense of smell that a dog would probably regard as a disability. Drop an
average modern human being naked into a demanding ecosystem and observe how
quickly the supposed master of the planet begins looking for a hotel.
What made us extraordinary was not physical supremacy but a peculiar combination
of abstraction, language, social cooperation, cumulative culture, tool-making and the
ability to transmit knowledge across generations. In other words, intelligence became
powerful because it acquired agency within a social system.
That may also be the more interesting question about AI. Not simply how intelligent it
is, but what agency we have given it: what systems can it control, what objectives
can it pursue, what resources can it access and what constraints remain outside its
control? A calculator has exceeded human beings at arithmetic for decades and
civilisation somehow endured. A crane is stronger than a human being, a jet is faster,
a microscope sees more, a telescope sees farther and a database remembers more.
Google Maps knows Basel better than I do and sometimes conveys this fact with a
degree of confidence I find unnecessary.
We did not interpret these technologies as rivals to humanity because we understood
them as extensions of human capability. The interesting question is therefore not
whether AI surpasses us at tasks; it already does. The interesting question is whether
superiority in tasks becomes autonomy of purpose. Capability is not the same thing
as sovereignty.
And then there are deepfakes. A few years ago, deepfakes were regularly presented
as the technology that would destroy our ability to distinguish truth from fiction. To be
clear, deceptive synthetic impersonation remains a serious problem: fraud, political
manipulation, fabricated evidence, extortion and reputational attacks are genuine
risks.
But something unexpected has also happened. Synthetic media became ordinary.
We encounter AI-generated voices, reconstructed images, altered photographs,

digital avatars and artificial video constantly. Not every AI-generated video is a
deepfake, of course; the term properly implies some form of synthetic impersonation
or manipulated representation. But the wider world of synthetic media has already
entered ordinary culture.
And society has begun developing antibodies. We ask where something came from,
platforms label synthetic material, authentication technologies evolve, journalistic
verification changes, courts adapt evidentiary standards and people become more
sceptical of astonishing videos—sometimes disastrously sceptical, admittedly. But
this is what civilisation does. Technology changes, fraud changes, trust changes,
verification changes, and then fraud changes again.
Humanity has been participating in this depressing little dance since the invention of
writing. The printing press enabled scholarship and propaganda; photography
created documentary evidence and photographic forgery; radio created mass
education and mass persuasion; television created journalism and advertising; the
internet created Wikipedia and your uncle’s WhatsApp group. Technology is rarely
morally coherent. Intent matters, institutions matter, incentives matter and users
matter.
Which raises another possibility: perhaps the danger lies not merely in technology but
in our love of apocalypse. Apocalypse is intellectually seductive, whereas moderation
has dreadful marketing. Nobody gets invited onto television to say, “This technology
will probably produce enormous benefits, several serious problems, some
unexpected consequences, a regulatory response of uneven quality and a difficult
social adjustment lasting approximately thirty years.” That person may be correct, but
they are also never getting a Netflix documentary.
Catastrophe has narrative structure. There is an invention, there is hubris, there is a
scientist nobody listened to, there is a warning, there is a machine that becomes
conscious at 3:17 a.m., and there is usually San Francisco being destroyed.
Civilisation ends shortly before the closing credits. Reality is annoyingly less
cinematic.
Most technologies neither save the world nor destroy it. They alter the distribution of
opportunities and dangers, create winners and losers, solve old problems while
creating new ones, amplify capacities already present in human society and,
eventually, get used by someone to deliver pizza.
Artificial intelligence may indeed turn out to be radically more consequential than
previous technologies. There are perfectly rational reasons to worry about
autonomous weapons, cyberattacks, biological design, fraud, mass surveillance,
labour displacement, manipulation, concentration of power and highly autonomous
systems operating at speeds humans cannot meaningfully supervise. These are not
imaginary risks and they deserve serious thought.
But taking danger seriously is not the same thing as assigning maximum probability
to maximum catastrophe. Sometimes a scenario is terrifying because it is plausible;

sometimes it is terrifying because human beings are extremely good at imagining
terrifying things. Distinguishing between the two is called risk assessment. Failing to
distinguish between them is called social media.
And this is where my contrarian argument encounters the version of me that actually
believes in caution. The lesson of Asilomar is not “do not worry.” The lesson is almost
the opposite: the scientists worried, paused, argued, created safeguards and then
proceeded. That sequence matters.
Asilomar did not stop biotechnology; it enabled biotechnology to continue under
conditions that society could increasingly understand and regulate. That may be the
precedent worth remembering for artificial intelligence. The answer need not be
panic, complacency, prohibition or worship, but something much less emotionally
satisfying: proportionate caution.
That means regulating capabilities whose failure could cause severe harm, restricting
autonomous access to dangerous infrastructure, demanding accountability when
systems exercise consequential power, improving authentication and provenance
systems, investing in safety research alongside capability research and remaining
alert to unknown unknowns. At the same time, we should resist the seductive
assumption that because a future cannot be predicted, it must therefore contain the
worst thing we can imagine. There is a difference between humility before uncertainty
and terror before uncertainty: one produces science; the other produces headlines.
Human beings have always been frightened by the moment when a tool begins to
exceed the limitations of its maker. The machine becomes stronger, the calculator
faster, the computer remembers more and the algorithm recognises patterns we
cannot see. AI can increasingly write, analyse, design, simulate, reason and create in
domains we once considered uniquely human. Perhaps that should humble us, but it
should not necessarily terrify us, because being surpassed is not new.
A crane can lift more than I can, a jet travels faster, a microscope sees things I
cannot see, a hard drive remembers more than I do, a cheetah runs faster, a
bloodhound smells better and a bat navigates in darkness. The existence of superior
capability elsewhere has never, by itself, diminished the distinctiveness of being
human. What matters is how that capability is situated within a system of agency,
intention, incentives and power. That is the boundary worth watching.
Perhaps there is also a deeper historical pattern here. Every generation tends to
believe that its technological crisis is unique because it is the first technological crisis
happening to us. The recombinant-DNA pioneers probably felt they were
approaching a boundary no previous generation had crossed. The lawyers
confronting Chakrabarty felt the legal categories of life and invention were breaking
apart. Dolly seemed to dissolve the distinction between reproduction and
manufacture. Human cloning appeared to threaten the uniqueness of individual
identity. De-extinction now challenges the apparent finality of extinction. Artificial

intelligence challenges another cherished boundary: the distinction between human
cognition and machine capability.
That does not mean the boundary is unimportant. It means boundary crises are not
new. Humanity repeatedly encounters a capability that destabilises an old category,
and the sequence is remarkably familiar: we panic, argue, legislate badly, legislate
better, normalise, and then discover a new boundary over which to begin the
argument again.
Perhaps artificial intelligence will truly be different. Perhaps this is the technology that
finally escapes every historical analogy, and perhaps the prophets of catastrophe are
right. But history should at least make us modest about the confidence with which we
announce uniqueness. We have heard versions of this announcement before.
The genetically altered bacterium did not escape into the drains and destroy
civilisation; patenting a bacterium did not lead to corporations owning the biosphere;
Dolly did not inaugurate factories full of cloned dictators; Hwang Woo-suk did not
produce the army of cloned human beings that fevered imaginations anticipated; the
great dam did not knock the Earth off its axis; and synthetic media did not make
reality disappear.
And somehow Homo sapiens—physically unimpressive, neurologically eccentric,
unable to outrun most animals and embarrassingly dependent on footwear—remains
here, modifying life, cloning sheep, editing genomes, discussing mammoths, building
artificial intelligences, becoming frightened of them, learning to live with them and
almost certainly beginning work on the next thing that will terrify us.
So yes, artificial intelligence deserves caution: serious caution, perhaps even
unprecedented caution. But every now and then, when someone announces that
humanity has finally invented the technology that will destroy humanity, it is
reasonable to remember our record.
The apocalypse has been announced before, and so far it keeps being postponed.