↓ Skip to main content
  1. writings/
  2. AI-Assisted Data Engineering/

Preface

·2 mins·
Table of Contents

What this book is
#

This is a practical account of how I actually work with AI agents as a data engineer, written so I can hold myself to it and so anyone doing similar work can borrow what fits. It is not a survey of every tool, and it is not a promise that the machine will do the thinking for you. It is a record of the habits, the setup, and the guardrails that let me hand real work to an agent and still stand behind what ships.

The through-line is that AI is an accelerator, not a substitute for judgment. It pays off when you already understand the problem well enough to verify the answer, and it becomes a liability the moment you let it run somewhere you cannot follow.

There is a personal reason I am writing it now. After more than a decade of building data systems, I want to take the next step in my own craft, the one that turns hard-won practice into something I can articulate clearly enough to be useful to someone else. Writing this down is how I make that move, from doing the work well to being able to teach it, and that ambition is as much a part of this book as any technique in it.

Who I am writing for
#

I am writing first for the engineer I was a few years ago, capable at the craft but new to letting a model touch the work. If that is you, this book is meant as company from someone a little further down the same road, not a lecture from someone who never made the mistakes.

It assumes you can already read a query, reason about a data model, and recognize when an answer is wrong. The faster you can verify, the more this way of working gives back.

How I will write it
#

I write in the first person because the choices and opinions presented here are most certainly mine. When the book turns toward setup or habit, I will often speak in the second person, because the reader I have in mind most often is still a version of me.

I will use concrete examples from real work, including the times an agent took a liberty I did not ask for and the recovery that followed. Tools and model names will change underneath this book, so I will keep the durable principles in the foreground and treat the specifics as illustrations, not the point.

Related

what room to breathe makes room for

i argued before that room to breathe protects me from burnout and helps me understand what i shipped, but that is only half of it. the space i refuse to fill with one more small improvement does not stay empty, it is where the slower, bigger ideas finally get room to take shape, including the longer thing i recently hinted at. room to breathe is not only rest, it is what makes space for something bigger to come.

room to breathe

ai removed the natural pause between improvements, so it always feels like there is one more optimization to squeeze out. but unless your system is fully autonomous, you are still the bottleneck, and your real rate of shipping work you understand has not changed. i argue that the fix is deliberate room to breathe, letting a solution run in production and settle in your head before you start the whole improvement loop again, both to avoid burnout and to actually understand what you built.

the guardrails i actually use with ai agents

in ai liberty i argued that capable models take more liberties and need robust guardrails, but i did not show them. this is the concrete kit i actually use to keep speed without losing control, namely read-only and ask modes, command allowlists, scoped database credentials, branch protection, and human-in-the-loop gates for the few actions that are genuinely hard to undo.

ai liberty

as ai models become smarter and more capable, they also become more confident, making them increasingly likely to take autonomous actions such as running terminal commands and executing sql injections without explicit instructions. i share recent examples of this behavior and discuss why robust guardrails are essential before unrecoverable damage is done.