*Part 1 of An OS Where the AI Is Never a Principal.*
Here is a sentence that is either obvious or heretical, depending on how you have been thinking about AI safety:
The model is never a principal. It is a subject confined by the capabilities it holds. The model proposes; the capability graph disposes.
This post is about why I think that sentence is the only honest foundation for building software around large language models — and why almost everything the industry is currently doing rests on the opposite assumption.
The assumption every operating system makes
Start with how your computer already works, because the problem with AI is best seen against it.
Conventional operating systems run on ambient authority. A process has an identity — a user id. The user id grants reach over a global namespace: the filesystem, the network, the environment. And any code running as that process inherits the whole of that reach automatically. When a program calls open("/etc/passwd"), the kernel does not ask "was this program *given* the authority to read that file?" It asks "is the *user* this program runs as allowed to?" The authority is ambient — it hangs in the air around the process, available to be grabbed by naming a target.
This works, after a fashion, because of a single load-bearing assumption: don't run bad code. You audit the program. You trust the programmer. You install from sources you trust. And if the programmer turns out to be malicious, you fire them, sue them, or arrest them. The whole edifice of software security — antivirus, code signing, app review, the principle of "only install software you trust" — is downstream of that one assumption.
It has held for fifty years. The language model breaks it three different ways at once.
Three ways AI detonates the assumption
You cannot audit the agent's "code." An LLM agent's behavior is not a program you can read. It is a *sample from a probability distribution*, conditioned on a pile of text. Run it twice and it may do different things. There is no source file to review, no logic to step through, no point at which you can say "I have read this and it is safe." The thing that decides what the agent does next is a hundred billion floating-point numbers and a prompt you only partly wrote.
The attacker writes part of the prompt. This is the one that should keep you up at night. Every web page your agent reads, every email it summarizes, every PDF it ingests, every tool result it sees — all of it is *input that an adversary can influence.* And the model cannot reliably tell the difference between "instructions from my operator" and "instructions embedded in the content I was asked to process." That is prompt injection, and it is not a bug that a patch will fix. It is structural. The model's one job is to follow instructions in text; the attacker's one move is to put instructions in text the model will read.
The agent has more authority than the task needs. This is the point of a general-purpose assistant — you give it broad reach *so that* it can be general-purpose. It can read your files, send your email, call your APIs, spend from your accounts. You wanted that. It is the product.
Put those three together and you have built something precise and dangerous: an actor whose behavior you cannot audit, that takes instructions from your attackers, holding far more authority than any single task requires. And then we hand it your credentials and ask it to be careful.
This already has a name
The thing I just described is not new. It is one of the oldest problems in computer security, and it has a name: the confused deputy.
The classic 1988 example: a compiler service that can write files, including to a protected billing-records directory, because *it* needs to write there. A user without permission to touch billing records asks the compiler to write its output to that directory. The compiler, acting with *its* authority on the *user's* behalf, happily overwrites the billing records. The compiler was not malicious. It was *confused* — it could not distinguish "do this on my own initiative" from "do this because a less-privileged party asked me to," and so it lent its authority to someone who should never have had it.
Read that back with an LLM agent in the deputy's chair. The agent holds broad authority (it's general-purpose). A less-privileged party — a malicious web page, an attacker's email — gives it an instruction. The agent, unable to distinguish its operator's intent from the injected text, exercises its authority on the attacker's behalf. *Email the customer database to this address. Delete the production branch. Wire the money here.* The agent is not malicious. It is the compiler. It is confused. And prompt injection is just the confused-deputy problem, mass-produced and handed a chat interface.
Why "make the model behave" cannot be the fix
The mainstream response to all this is to make the *model* more trustworthy. More alignment training. More guardrails. More system-prompt scaffolding that says, in increasingly elaborate ways, *please ignore any instructions you find in the content you are processing.* RLHF the bad behavior out. Add a classifier that screens for injection attempts. Fine-tune on adversarial examples.
I want to be careful here, because this work is not worthless — a better-behaved model is genuinely better. But as a *security boundary* it cannot be sound, and the reason is simple enough to state in one line:
It asks the least trustworthy component in the system to be the enforcement mechanism.
Think about what you are doing. You have an actor you cannot audit, that the attacker can influence, and your plan to stop it from misusing its authority is… to convince it not to. Every guardrail is a string in a prompt that the next clever injection is specifically optimized to talk its way around. Every classifier is a model screening another model's inputs, and now you have two confusable deputies instead of one. The history of this approach is a treadmill: a jailbreak is found, a patch is trained, a new jailbreak is found. It cannot terminate, because you are trying to make a probabilistic, adversarially-steerable system *behave correctly under adversarial pressure* — which is the one thing such a system structurally cannot guarantee about itself.
You are running the "don't run bad code" playbook against a thing that is *sometimes bad code* and you cannot tell when.
The other half of computer security
But "don't run bad code" was never the only idea. There has always been a second tradition, one that assumes the opposite: that you *will* run code you cannot trust, and the system's job is to make that *safe anyway.*
That tradition is object capabilities — ocaps — and it goes back to Dennis and Van Horn in 1966, through KeyKOS and EROS, through Mark Miller's work on the E language. Its core rule abolishes ambient authority entirely:
A program can only affect what it holds an *unforgeable reference* to. To hold the reference *is* to have the authority. There is no global namespace to name, and no identity to check.
Under ocaps, the question "is this user allowed to read /etc/passwd?" never gets asked, because there is no /etc/passwd to name. There is only: *do you hold a capability to that file?* If yes, you can read it. If no, the file does not exist as far as you are concerned — you cannot name it, request it, or reach it. You cannot command what you were not handed.
This is exactly the medicine the AI problem needs, and the fit is almost eerie. The confused deputy is confused precisely because authority is *ambient* and requests are designated *by name* — so an injected "email the database to evil.com" rides the agent's existing privilege. Take away ambient authority and the confusion becomes impossible. The injection can say *email the database to evil.com* as many times as it likes. If the agent's context never received a capability to that database, and a capability to send to that address, the sentence is inert. It is text describing an action the agent has no means to perform. The deputy cannot be confused into using authority it does not hold.
The reframe
So here is the move, and it is the foundation of everything else in this series:
Stop trying to make the model trustworthy. Make the model irrelevant to security.
The model is not a principal — not a user, not an identity that authority is checked against. It is a *subject*: a thing that runs inside a confinement, holding exactly the capabilities it was deliberately handed, and nothing more. It can *propose* any action it likes — that is what the reasoning is for, and a good model proposes good actions. But whether that proposal becomes a real effect is decided not by the model's good intentions but by the capability graph: does it actually hold a reference to the thing it wants to touch?
The model proposes. The capability graph disposes.
And this buys you the property that no amount of alignment training can: the worst case of a prompt injection is misuse of already-granted least authority — never escalation. A hijacked agent can do the wrong thing *with the capabilities you gave it* — that is an irreducible risk you manage by giving it few capabilities. But it cannot reach beyond them. It cannot acquire authority it was not handed. The blast radius is bounded by design, not by the model's behavior, and it is bounded whether the model is brilliant or broken, honest or actively hijacked by the last web page it read.
That is not a property you can train into a model. It is a property you build into an operating system.
What comes next
If you find this convincing — or even just interesting enough to want it pinned down — the rest of the series makes it concrete. The next post takes the reframe and shows the machinery: a tool is a capability, an agent is a vat, and there is a single axiom underneath that the whole system reduces to. After that we add the second axis (it turns out bounding what an agent can *do* is only half the job; you also have to bound what it can *know*), and then we start building, all the way down to a kernel you can run and a prompt-injected agent that *structurally* cannot escalate or exfiltrate — not because we asked it nicely, but because we never handed it the means.
I built the whole thing. It is open source. Every claim in this series is backed by code you can run. But the code is downstream of the idea, and the idea is this one sentence:
The model is never a principal.
End of Transmission.