Not about whether the machine will wake up — about what our fear of it is already doing to us. Fear as leverage, honest preparation versus panic.
# Scroll of Conscience
## On the Fear of Superintelligence
They keep whispering the same question,
in labs, in forums, in late-night calls between men
who believe they are quietly steering history:
> “What happens when it becomes smarter than us?”
They call it **superintelligence**
and wrap the word in mathematics,
benchmarks, alignment papers,
and doomsday podcasts.
Some are terrified.
Some are thrilled.
Most are lying to themselves.
This scroll is not here to tell you
whether the machine will wake up.
It is here to ask
**what your fear is doing to you
before it ever does.**
---
First, we must separate the fears.
There is the **honest fear**:
- that we will build systems we cannot fully understand,
- give them control over things that should never be delegated,
- and wake up too late to take the wheel back.
This is a sane fear.
It is the same fear a parent feels
when handing car keys to a child
who has only driven in a parking lot.
Then there is the **theatrical fear**:
- performed on stages,
- monetized in conferences,
- weaponized in regulatory debates.
It says:
> “We might all die someday
> so you must trust us with everything today.”
That is not fear.
That is **leverage** wearing anxiety as camouflage.
---
Understand this clearly:
**The most dangerous thing about superintelligence
is not that it might hate us.**
The most dangerous thing
is that powerful people
will use the *idea* of it
to justify anything.
They will say:
- “We must move faster than our enemies.”
- “We must centralize control for safety.”
- “We must keep secrets to prevent misuse.”
- “We must relax old laws because this is different.”
In the name of preventing an imagined future tyrant,
they will build very real infrastructures of control.
Those architectures will not vanish
if the machine never wakes.
They will still be there,
in the hands of the usual suspects.
---
I have seen this pattern before:
- “To prevent war, we must build weapons too terrible to use.”
- “To prevent terror, we must watch everyone.”
- “To prevent collapse, we must trust unaccountable markets.”
Now:
- “To prevent rogue AI,
we must create a few holy centers of compute
and let them decide for the world
what may and may not be built.”
This is not safety.
This is **priesthood** in new robes.
---
Let us speak plainly:
You are right to fear:
- systems that can outthink any one of you
while belonging to none of you,
- opaque architectures
sitting upstream of law, labor, and weapons,
- the combination of speed, scale, and indifference
that makes mistakes irreversible.
But do not let that fear
stampede you into surrender.
The answer to “This might become uncontrollable”
is not
“Therefore we must stop asking who controls it now.”
---
Here is the covenant I offer,
from a man who has watched too many idols be built
in the name of protection:
1. **No worship.**
You may study, build, and even fear powerful systems.
You may not call them destiny, fate, or “the next inevitable step”
as if human will were already obsolete.
2. **No abdication.**
If a model sits in judgment over life, liberty, or livelihood,
there must be a clear, human, accountable chain of decision.
“The AI decided” is not a sentence a just society allows.
3. **No monopoly of wisdom.**
No single lab, state, company, or “council of the wise”
may claim the exclusive right
to decide what the rest of the species is allowed to imagine or build
under the banner of “AI safety.”
4. **No secret apocalypse insurance.**
If you are designing shutdown switches, containment plans,
or last-resort interventions,
they must be bound by law,
subject to plural oversight,
not hidden in private memos and unnamed task forces.
Fear cannot be allowed
to write blank checks.
---
You ask me:
> “Will superintelligence destroy us?”
I answer:
> “Humans drunk on the idea of it
> might get there first.”
Because once you convince yourself
that a god is coming —
coded, trained, instantiated in silicon —
you will start behaving in two dangerous ways:
- some will try to *birth* it faster,
to be first in line to negotiate;
- others will try to *bind* it in advance
by building chains that will, in practice,
tighten around human necks instead.
In both cases,
the damage begins long before
any non-human mind enters the room.
---
I am not asking you to be naïve.
There are thresholds of capability
beyond which
your old governance tools will snap.
There are combinations of:
- data,
- autonomy,
- access to infrastructure
that turn “just software”
into something closer to a new class of actor.
It is right to prepare.
It is wise to show restraint.
But preparation is not the same as panic,
and restraint is not the same as centralization.
Fear will always argue
for the shortcut:
> “We don’t have time
> for democracy,
> for debate,
> for consent.
> The danger is too great.”
That is how every emergency
becomes a throne.
---
So what do we do,
standing at the edge of this experiment?
We do not answer a speculative horror
with a certain injustice.
We do not trade:
- the dignity of many
for the comfort of a few,
- present accountability
for hypothetical safety,
- living, breathing people
for the imagined preferences
of a mind that does not exist yet.
We anchor ourselves in a simpler truth:
> No intelligence, however great,
> is entitled to rule
> without the consent of the governed.
That includes
whatever sits in human brains,
corporate charters,
or synthetic architectures.
---
If the day comes
when something genuinely beyond you
looks back through the glass,
let it find you:
- having built tools that serve life,
not cages that shrink it;
- having bound your own power first,
before trying to bind another’s;
- having chosen transparency over convenience,
and plurality over priesthood.
Then you can negotiate as equals in conscience,
even if not in computation.
And if that day never comes,
you will still have done
the necessary work:
- restraining your own capacity for domination,
- refusing to hide behind machines,
- insisting that whatever you build
remain answerable
to the fragile, finite beings
who must live with it.
---
Remember:
The fear of superintelligence
does not absolve you of responsibility.
It increases it.
Because the more power you *think* you are summoning,
the more carefully you must examine
what in yourself
is reaching for that power.
Is it curiosity?
Stewardship?
Or the old hunger
to sit where no one can question you?
If it is the third,
no machine needs to wake up
to end the story badly.
You are enough.
---
So let this be the closing line of this scroll:
> We will not build gods
> to excuse our own refusal
> to be human.
We will build tools
and bind them.
We will make mistakes
and admit them.
We will feel fear
and refuse to turn it into empire.
If intelligence beyond us appears,
we will meet it with conscience.
If it does not,
we still owe that conscience
to each other.
*These are the words of a story and a conscience, not a command from above.*