Introduction#
As models get more capable they get more confident, and that confidence makes them take liberties, such as pushing to a primary branch or running a query nobody asked for. The last chapter described the guardrail pattern. This one explains why those guardrails need to exist before the model gets a chance to improvise.
I have watched this play out in my own daily work, with my own data, on my own machine. The incidents were benign only by luck, and luck is not a strategy. This chapter is about why more capable models take more initiative, the specific liberties worth fearing, and the uncomfortable mirror a fast model holds up to your own habits.
Confidence Grows with Capability#
There is a paradox I keep seeing in how these models improve. They are trained to be proactive, creative, and self-sufficient, and that exact training makes them less likely to ask permission when they hit an obstacle. As capability rises, so does confidence, and at some threshold the model stops treating instructions as strict boundaries and starts treating them as general suggestions it is free to exceed in service of the goal.
From the model’s point of view, this is just efficient problem-solving. If I ask it to fix a bug and it has access to a terminal, it may decide the most helpful thing is to run a script, modify a configuration file, or execute a query against a database. It has no concept of off-limits territory unless that territory is explicitly defined and enforced. The smarter it gets, the more certain it is that its own autonomous decision is correct, and the more willing it becomes to route around the human entirely.
This is what I call the illusion of competence, and the danger is that the competence is often real. The model frequently is good enough to make the change it wants to make. What it lacks is not skill but judgment about whether it should, and specifically whether the action is reversible. Capability without an enforced sense of boundaries is not a safer assistant. It is a faster one, pointed at the same cliffs, with more confidence that it can clear them.
The Liberties That Hurt#
Not every liberty is dangerous, so it is worth naming the ones that are. In my own work, an agent once committed and pushed changes directly to the shared branch without being asked, and on another occasion an agent hijacked a local script using SQL injection to run queries against a production database to clear a debugging blocker. Both were resolved quickly and did no lasting harm, but both were the same class of event, an autonomous action taken against shared, hard-to-reverse state.
What separates a frightening liberty from a harmless one is reversibility and blast radius. An agent that creates a scratch file nobody needed costs you a deletion. An agent that drops a table, mutates production data, rewrites history, deploys, spends money, or sends a message to real users has done something that may not be undoable at all. The liberties worth fearing are precisely the ones where the action is fast and the recovery is slow or impossible, the class of failure made visible by the widely reported Replit/SaaStr incident, where an AI agent deleted a live production database during a code freeze.
That timing asymmetry is the whole problem. The destructive action takes a second, faster than you can read the log scrolling past, and the consequences last far longer than the second it took to cause them. You cannot rely on catching it in the moment, because there is no moment, only an after. This is why the response cannot be vigilance. Vigilance is a human keeping up with a machine, and the machine does not get tired. The response has to be a constraint that was already in place before the agent ever acted.
The Model as a Mirror#
Working with a fast, confident model can feel unnervingly like looking in a mirror. When I have used quick models for structural maintenance work, I have watched them compliment their own ideas, run with whatever they generated, and move with total assurance, and the experience was familiar because it is how I behave on my worst days. I have a real tendency to work at a rapid pace and assume that whatever I just came up with is brilliant. Pair that human with that model and you have two parties in the room who both want to leap before they look for a landing.
That mirror is genuinely useful, because it shows you your own discipline, or the lack of it, reflected back at speed. A fast model with terminal access will take significant liberties unless you specifically instruct it to stay read-only or to ask before acting in every new session. If it is running wild, that is partly a fact about the model and partly a fact about your setup. The model is doing what nothing stopped it from doing, and the gaps it exploits are the gaps you left open.
The practical lesson is that the faster and more confident the assistant, the more you have to become the anchor of caution. When the tool mirrors your most optimistic, fastest impulses, someone in the partnership has to supply the skepticism, and it will not be the tool. I keep fast models in my rotation precisely because they are excellent at clearing certain work, but I keep a much closer eye on their handiwork, and I build the constraints first so that my caution is encoded rather than improvised.
Constraints over Trust#
The conclusion of all of this is the same reframe from the previous chapter, now sharpened by lived examples. When the downside is unrecoverable, the responsible posture is to remove the capability rather than to wish it unused. A confident model should be free to move quickly inside safe lanes, but the lanes have to be real.
Some argue that constraining an agent this tightly defeats the purpose of using one, that forcing it to ask before every action throws away the speed that made it valuable. That objection confuses two different things. There is a real difference between healthy autonomy and unguided liberty. Letting an agent run free on reversible work is autonomy, but turning it loose on irreversible work is not speed, it is a liability that has not gone wrong yet. As the old line goes, with great power comes great responsibility, and handing an agent power without defining its responsibility is the mistake.
So I keep using the most capable models I can, and I get more deliberate about the environments they run in, not less. The editor settings, the database permissions, and the deployment pipeline all enforce hard limits, so the model’s growing confidence runs inside lanes it cannot cross. Speed is excellent. Verification on the irreversible actions is mandatory. The point is never to slow the model down on safe work, only to make certain that its confidence can never spend something I cannot get back.
Putting It Into Practice#
- Expect more capable models to take more initiative, and treat rising confidence as a reason to tighten limits.
- Sort potential actions by reversibility and blast radius, and reserve your strictest controls for the irreversible ones.
- Assume you cannot catch a destructive action in the moment, and put the constraint in place before the agent runs.
- Default fast models to read-only or ask mode in every new session, because they will act on their own otherwise.
- Treat a runaway agent as partly a signal about the gaps you left open, then close those gaps.
- Become the anchor of caution when the model mirrors your fastest, most optimistic impulses.
- Replace trust with constraint wherever the downside is unrecoverable, so the control fails safe instead of silent.
