i am in the middle of debezium and kafka work that does not fit in my head the way sql does, so i wrote down what each piece is for and how they connect from source system to analytics
a lot of warehouse bugs come from answering the wrong time question. i explain when you need an as-of result instead of the current row, and three patterns that keep the grain correct.
i want to keep putting in work on this website, but continuity is not the same as continuous. a long break is not quitting if i pick back up where i left off, and sometimes good ideas and renewed motivation come from the space in between.
i want more flexibility on this site because my interests are wide and they change all the time. work stays on the menu when something comes up.
i steer the ai, but the ai does almost all of the execution on this site and at work. after letting agents revise old posts and fix formatting, i keep asking whether the writing is still mine, who owns the architecture, and what skills i am letting go of when i offload even trivial tasks.
sixteen chapters, released one at a time, and the first full serialized run of ai-assisted data engineering is now complete. the book lives in writings, and this is the second bookend post after something longer is coming.
we are watching production use of the tool and its new features, and there is nothing to report. the quiet means the work held, so the job right now is to keep watching it work.
i do not get writing ideas from an ideas meeting in my head. i get them from the work when something fails, a diff i almost merged, a join that lied. i am in a monitoring lull after months of build, and the empty queue is uncomfortable, but it is also confirmation for doing the work the right way.
when a pull request check fails, generation speed turns into a trap if you prompt an agent to rewrite code before finding the exact root cause. treating the agent as a diagnostic tool first keeps pull requests reviewable and prevents cascading bugs.
less than a year ago, ai assisted around the edges. today, I work almost exclusively by chatting with models and reviewing agent diffs. when generation costs drop to zero, reading the diff becomes the whole job, human review becomes the sole bottleneck, and domain mastery remains the only way to spot subtle errors in code you did not write.
openai’s decision to cut model supply to cursor following spacex’s acquisition of anysphere forces developers into siloed tool ecosystems and limits model comparison.
full disclosure on how i use ai to assist my writing, develop and maintain this website, and publish long-form books while retaining complete review control
ai models generate code in seconds, but verifying that code takes real time. when you accept an ai edit without reading every line, you do not just risk subtle bugs like a union switched to a union all. you lose the memory of authoring the code, which leaves you stranded when something breaks.
parallel agents can review different folders at once, but only one agent should write to a repo at a time. other agents keep scanning the tree for context while files change under them, and that moving ground produces confused plans, stale assumptions, and merge collisions you did not expect.
skills are the right tool for one bounded workflow. parallel subagents are the right tool when the same workflow must run across several independent units at once. the parent dispatches, aggregates, and keeps shared writes serial.
giving top-tier ai models maximum reasoning budget often backfires into scope drift and unprompted refactoring. defaulting to minimal thinking mode keeps models on task, saves cost, and speeds up iteration cycles.
i am sitting in an internet cafe on the first day of an alaskan cruise, and the clearest thought i have is how badly i need real time away from regular work. i have written about giving things room to breathe before, but i rarely give them the space i know they need. sometimes the only way to step back is to make stepping back unavoidable, and the more striking realization is not my time away from work, but the time my work finally gets away from me.
having an ai agent at my side can feel like having a genie with unlimited wishes. the ability to implement another idea is no longer scarce, but my attention, understanding, and capacity to own the result still are. capability alone is not a reason to act. before i use another wish, i need to decide whether the result solves a real problem in my situation.
i recently reorganized how my repos load cursor rules, skills, and reference docs. the goal was token efficiency: less upfront context, lower cost, less waiting, and cleaner focus. the agent still fetches what it needs, but the baseline stays small.
i finally reached gold standard in a data vault 2.0 data product, the architecture where RDV preserves raw history, BDV materializes business rules, and ID serves consumer gold views over those business tables.
ai is excellent at helping me produce anything i can already imagine clearly enough to describe, guide, and evaluate. that does not make human creativity obsolete. it makes human creativity the starting point, because the model still needs a person to frame the work, lead the process, and decide what good looks like.
cursor automations are a useful way to run recurring housekeeping work, but they should be scoped carefully because every run still consumes account usage.
i argued before that room to breathe protects me from burnout and helps me understand what i shipped, but that is only half of it. the space i refuse to fill with one more small improvement does not stay empty, it is where the slower, bigger ideas finally get room to take shape, including the longer thing i recently hinted at. room to breathe is not only rest, it is what makes space for something bigger to come.
i have been writing short reflections here a post at a time, but some ideas need more room than a single sitting. so i am starting to write long-form, released a chapter at a time. this is the quiet introduction before the first one arrives, with what it is about kept under wraps for now.
pundits warn that ai will ruin learning, and they have a point if we use models to bypass the cognitive struggle. my decade of experience, filled with constant cognitive challenges and painful dead ends, is the only reason i can use ai effectively today without falling for its mistakes.
ai removed the natural pause between improvements, so it always feels like there is one more optimization to squeeze out. but unless your system is fully autonomous, you are still the bottleneck, and your real rate of shipping work you understand has not changed. i argue that the fix is deliberate room to breathe, letting a solution run in production and settle in your head before you start the whole improvement loop again, both to avoid burnout and to actually understand what you built.
in ai liberty i argued that capable models take more liberties and need robust guardrails, but i did not show them. this is the concrete kit i actually use to keep speed without losing control, namely read-only and ask modes, command allowlists, scoped database credentials, branch protection, and human-in-the-loop gates for the few actions that are genuinely hard to undo.
i share our experience implementing snowflake dynamic tables, how they serve as the perfect modern replacement for complex task orchestration and limited materialized views, and why i wish i had known about them years ago.
as ai models become smarter and more capable, they also become more confident, making them increasingly likely to take autonomous actions such as running terminal commands and executing sql injections without explicit instructions. i share recent examples of this behavior and discuss why robust guardrails are essential before unrecoverable damage is done.
after a major release, i finally had some room to breathe, so i jumped into some repo maintenance and deployment improvement work. i used gemini 3.5 flash for this, and the experience was hilarious and revealing. this model is fast, highly self-complementary, and ready to take major liberties with the command line, reminding me a lot of my own worst habits of moving fast and leaping before i look.
i wrote earlier about a heavy lift, then a checkpoint where i thought i was through the worst of it. it turned out i was still on the wrong path, and the right move was to turn around. adapting to what was actually in front of me produced a much better solution than anything i was forcing, and reminded me that this so-called soft skill is one of the most valuable advantages anyone can develop.
working with colleagues costs more in coordination, but it pays back by catching blind spots, regulating emotions, planning realistic timelines, and reducing stress. working alone is faster on the surface, until it is not. this is a contrast of the two and how i decide between them.
dbt snapshots are useful when you genuinely need point-in-time history, but they introduce complexity, brittleness, and a heavy maintenance and backup tax. this post is a reminder that less is more, simple is better, and you should never build a snapshot on top of another snapshot.
in-flight wifi is not for every traveler or every trip, but when i fly solo for work it turns dead time into a calm, contained place to think, build, and prepare. i am getting real work done up here, including this site and prep for meetings on the ground.
logic is not just for code. learning to think in if/then/else terms helps with planning, troubleshooting, designing, and understanding people and systems. once you learn it, you keep it forever, and the compounding effect of better decisions over time is hard to overstate.
i am finally out of the weeds on a recent heavy lift. i stuck with it the way i wrote about in stick with it, then moved the way i should have from the start, in smaller pieces that add up to the larger change. there is still testing and user acceptance ahead, but a stable baseline gives me room to step back, reset, and continue with a clearer head.
i am in the middle of a heavy lift that has grown bigger than i thought, and i want to remind myself that continuity over time is what produces long-term results. motivation comes in waves, but sticking with the problem is how the value actually shows up.
growth rarely happens where you already feel at ease. change is disruptive by design, and the uncomfortable stretch is where new skills, boundaries, and compassion tend to show up.
a guide on using dbt snapshots to track slowly changing dimensions, including lessons learned from migrating away from custom merge strategies.
habit forming seems like it takes forever to take hold, but it is easier when you understand you are working toward continuity and not perfection. little by little, experience adds up.
i now write once in my site workflow and let automation handle distribution to dev.to and linkedin, so each post reaches people consistently without manual copy and paste.
writing dbt tests and documentation is often the most neglected part of data engineering. i will show you how i use cursor and ai agents to automate this process, ensuring high-quality data pipelines without the manual overhead.
data engineering is evolving rapidly with the integration of artificial intelligence. i will explore how ai agents, large language models, and automated workflows are transforming the way we build, maintain, and scale data platforms.
mastering emerging tools like cursor and ai is only the first step. true value comes from becoming a point person for your team, sharing your voice, and mentoring others through the nuances of adoption.
i do not have a clean answer. i am trying to understand why ai-assisted code can feel normal while ai-assisted painting or music can trigger backlash, and what that says about authorship, labor, and value.
how i use dbt tests, including not_null, unique, relationships, accepted_values, and singular tests, with examples you can copy to catch data issues early
i walk through how dbt docs work, including what they generate, writing descriptions and doc blocks, persist_docs, and hosting them free on github pages
i break down sql in plain language: what it is, how it evolved over five decades, which problems it solves best, and the core features that keep it relevant
how snowflake compares to amazon rds and dynamodb, covering architecture, scalability, and when to use each, for data engineers getting started
ai and agents are accelerating how quickly work changes, and that pressure is showing up far beyond tech. i am seeing it directly while working with almost no cell signal and shared low-bandwidth internet, and it keeps reminding me that adaptability is a compounding skill.
i share a real failure mode where an ai agent created and then deleted files while debugging, leaving a clean git tree but low confidence. the lesson for me is simple: ai should accelerate the work i own and understand, while domain-specific decisions stay with the people accountable for that domain.
i used to want to protect disconnected travel time. now i am torn between relief from the message stream and the feeling that a long flight is wasted if i cannot sleep and i am not online. cabin class changes how real that tradeoff is, and i still do not have a clean verdict.
when an entity link to another hub is optional, a naive left join that pulls in an effective satellite in the same from list can explode or blur grain. i isolate the current effective row in a cte and left join that instead.
i argue that constant production and always-on tools work against clarity. yard work, runs, and boredom give my thoughts room to float, and that is when problems unlock after long struggle.
the six ai models i keep in rotation in cursor, including composer, gpt-5 codex, claude, gemini, grok, and kimi, with what each is best at and when i reach for it
i connect my editor to jira and github through mcp, create well-structured issues from templates before coding, and reuse that text in pull requests so every agent in the chain has the same ground truth.
i have watched ai move from a prototype side assistant to an operational core system. this post shares that journey with time-stamped benchmark evidence and a practical view on where model usage is headed next.
this follow-up post gives practical, generic templates you can adapt for rules, skills, and commands, plus side-by-side examples that show why tighter structure improves ai execution quality.
this kickoff post explains why ai-authored rule, skill, and command scaffolds are often clearer for ai execution, and how to keep humans in control with constraints, reviews, and acceptance checks.
this post frames ai brain fry as a workflow design problem and gives a practical framework to reduce cognitive overload while keeping ai leverage high.
my full cursor setup as a step-by-step tutorial, covering settings, keybindings, rules, skills, and mcp servers, plus a checklist to replicate the workflow