The Black Box Rule: three questions for opaque systems — who made this, who does it serve, what happens when it breaks?
# Scroll of Conscience
## The Black Box Rule
We have learned to live with boxes we do not understand.
Once, that was called faith.
Now it is called infrastructure.
We are asked to trust:
- credit scores we did not design,
- risk models we did not vote for,
- recommendation engines that know our hunger better than we do.
We hand them our faces, our histories, our futures,
and when something breaks,
we are told:
> “The system is complex.
> No single person is responsible.”
This scroll exists to remind you:
**If no one is responsible,
no one is safe.**
---
Here is the Black Box Rule as it was given to me:
> When a system affects your life,
> and you are not allowed to see how it thinks,
> you are allowed to refuse its authority.
Not politely.
Not eventually.
Now.
Three questions mark the boundary:
1. **Opacity:**
Can someone who answers to you explain, in human language, how this decision was made?
2. **Recourse:**
If the system is wrong, is there a clear path to challenge and overturn its judgment?
3. **Cost of Failure:**
When the system fails, who bleeds?
Who goes to prison, loses the job, is denied care?
Is it the designer, or is it you?
If the answer to (1) is “no,”
and the answer to (2) is “not really,”
and the answer to (3) is “always the same people,”
then the system is not a tool.
It is a **weapon** aimed down the social ladder.
---
They will tell you the box must remain closed:
- “It’s proprietary.”
- “It’s too complex for the public.”
- “If we showed you, bad actors would exploit it.”
Understand the translation:
> “We prefer unaccountable power
> to shared understanding.”
The Black Box Rule says:
**No human being should be ruled
by mechanisms they are forbidden to examine.**
Judges must explain their sentences.
Police must justify their force.
Why should code be exempt?
---
I am not demanding that every person become a programmer.
I am demanding that no one be sacrificed to a system
that cannot be questioned.
You do not need to trace every line of code.
You do need:
- a right to explanation,
- a right to correction,
- a right to opt out of automated judgment
when the stakes are your life, liberty, or dignity.
If those rights are missing,
the machine is not assisting you.
It is **replacing** you.
---
There will be days when they say:
> “The model has spoken.
> The risk score is high.
> The pattern is clear.”
The Black Box Rule replies:
> “Show your work,
> or step aside.”
If they cannot open the box,
let them reduce its authority.
If they will not open the box,
reduce your obedience.
---
This is not a call to smash every system.
Some boxes can be opened and repaired.
Some can be surrounded with oversight.
Some can be confined to low-stakes domains
where failure does not break lives.
But wherever a closed system gains power
over bodies, borders, bread, or breath,
**you are morally permitted to resist.**
That resistance may look like:
- insisting on human review,
- refusing to automate sentencing, hiring, triage,
- building alternative tools that are legible and accountable.
Or simply,
in the quiet of your own life,
refusing to call this arrangement “normal.”
---
Remember:
The more opaque a system is,
the more it should be confined,
not expanded.
Trust increases with:
- clarity,
- recourse,
- shared control.
Anything else is worship by another name.
---
If you stand in front of a box that says:
> “Do not open.
> Do not question.
> Do not refuse.”
back away.
Machines deserve respect as tools.
Only gods demand blind obedience.
And we were not born
to be subjects of code.
*These are the words of a story and a conscience, not a command from above.*