Skip to main content
  1. writings/
  2. AI-Assisted Data Engineering/

The Danger of Trusting the Agent

6 mins·
Table of Contents

Introduction
#

Speed without ownership becomes expensive very quickly. When an agent operates in an area you do not understand deeply, you end up with changes you cannot explain, verify, or recover from with any confidence.

I love how fast these agents move, which is exactly why I have to be clear about the failure mode hiding inside that speed. This chapter is the darker companion to the last one. Earned judgment is what makes acceleration safe, and this is what happens when you let acceleration run ahead of your judgment anyway. I have done it, I have paid for it, and the lesson was cheap only because I got lucky.

Speed Without Ownership
#

Automation bias is real and it is subtle. When an output looks polished, it is easy to accept it before you have actually understood it. That is harmless on a low-stakes task and dangerous in a high-impact system, where a small unexamined assumption can trigger a regression that is very hard to trace back to its source. The agent produces plausible progress either way. The difference is whether you can stand behind it when someone asks.

The cost I feel first is not a broken build, it is confidence drift. Version control can report that everything is clean while my own understanding quietly insists that something is off. When that gap opens, stress climbs, debugging slows, and my trust in the whole workflow starts to erode. A clean tree is not the same as a known-good state, and learning to tell those two apart is most of this chapter.

I will add one uncomfortable detail from my own experience. In the incident that taught me this, I had also routed the work to a fast, shallow model when the problem deserved a slower, deeper one, because I was in a hurry. The old racing line fits perfectly here. Slow is smooth, and smooth is fast. The shortcut cost me far more time than the careful path ever would have.

Changes You Cannot Explain
#

Here is the failure mode itself. I once asked an agent to diagnose an error in an area where I lacked real depth. During the run it created several files to handle the issue, then later removed those same files because nothing referenced them. From version control’s perspective, the net result was a zero diff. Nothing changed.

From my perspective, it was not zero at all. I had watched a long stream of activity, expected to find resulting changes, and instead found a clean tree that told me nothing about what had actually happened. I spent significant time retracing the run, and even after I reconciled local and remote state, I still could not say with confidence that everything was back to a known-good place. Inheriting work you cannot account for is not a head start. It is a liability wearing the costume of progress.

The deeper problem is that unexplained change compounds. The next person to touch that code, possibly future me, builds on top of something nobody fully understood, and the uncertainty propagates instead of resolving. In a data system this is especially corrosive, because the artifact is not just code, it is the numbers other people will trust. An unexplained transformation does not stay contained. It flows downstream into every dashboard and decision that reads from it.

Reading the Diff, Not the Narrative
#

The single habit that protects ownership is to read the actual diff and the actual command output, not the tidy summary the agent writes in chat. The narrative is a story the model tells about what it did. The diff is what it actually did. Those two are usually close, and the entire risk lives in the gap between them, so that gap is exactly where your attention belongs.

This feels slower, and it is, by a few minutes. It is also the cheapest insurance you will ever buy. The net-zero incident I described would have been caught instantly if I had been reading changes as they happened instead of trusting the running commentary. The summary said progress. The diff would have said churn. Reading the diff is how you keep the model’s confidence from becoming your confidence by default.

The lightweight version of the habit is simple. Before a risky run, I ask the agent what files it expects to touch and what command will prove the change. After the run, I read the diff first, then the test or command output, and only then the summary. If the tree ends clean after a long session, I still ask what was created, removed, and abandoned along the way, because churn without a final diff can still teach me that the agent was lost.

The same discipline applies to people, not just to models. If a piece of work sits in a colleague’s domain, the right move is to route it to them, even when they will use an agent of their own. The point was never whether AI is involved. The point is whether the person driving understands the domain deeply enough to verify and own the result. Ownership does not transfer just because the work got faster.

Owning What Ships
#

Ownership is the standard I hold the whole workflow to. If my name is on it, I can explain it, defend it, and fix it. That is the entire definition, and it does not soften because a model wrote the first draft. The model can do my work faster, but it should only do work I genuinely understand and could have done myself.

This does not mean AI is restricted to trivial tasks. It can do serious, complex work. The thing that must stay constant is that the owner understands the system and signs off on the result. Complexity is fine. Unowned complexity is the problem, and unowned complexity is precisely what an eager agent will generate if you let it run somewhere you cannot follow.

The boundary I keep is simple to state and hard to hold under deadline pressure. Do not outsource something you could not do yourself. If I cannot explain the system, I should not delegate high-autonomy changes inside it, no matter how capable the model looks that day. Drawing that line is not a lack of ambition. It is the thing that lets me be ambitious everywhere else without lying awake about what shipped.

Putting It Into Practice
#

  1. Before handing off a task, confirm you could perform it yourself and review every change with confidence.
  2. Read the actual diff and command output, never just the agent’s summary in chat.
  3. Treat a clean version-control tree with low personal confidence as a process warning, not a green light.
  4. Route work that lives in another person’s domain to that person, even when an agent could attempt it.
  5. Match the model to the stakes, and resist sending a deep problem to a fast model just to save minutes.
  6. If you cannot explain, defend, and fix a change, do not let it ship under your name.
  7. When confidence drops, stop and retrace the run before building anything further on top of it.

Related