i am in the middle of debezium and kafka work that does not fit in my head the way sql does, so i wrote down what each piece is for and how they connect from source system to analytics
sixteen chapters, released one at a time, and the first full serialized run of ai-assisted data engineering is now complete. the book lives in writings, and this is the second bookend post after something longer is coming.
i do not get writing ideas from an ideas meeting in my head. i get them from the work when something fails, a diff i almost merged, a join that lied. i am in a monitoring lull after months of build, and the empty queue is uncomfortable, but it is also confirmation for doing the work the right way.
when a pull request check fails, generation speed turns into a trap if you prompt an agent to rewrite code before finding the exact root cause. treating the agent as a diagnostic tool first keeps pull requests reviewable and prevents cascading bugs.
i finally reached gold standard in a data vault 2.0 data product, the architecture where RDV preserves raw history, BDV materializes business rules, and ID serves consumer gold views over those business tables.
i share our experience implementing snowflake dynamic tables, how they serve as the perfect modern replacement for complex task orchestration and limited materialized views, and why i wish i had known about them years ago.
dbt snapshots are useful when you genuinely need point-in-time history, but they introduce complexity, brittleness, and a heavy maintenance and backup tax. this post is a reminder that less is more, simple is better, and you should never build a snapshot on top of another snapshot.
a guide on using dbt snapshots to track slowly changing dimensions, including lessons learned from migrating away from custom merge strategies.
data engineering is evolving rapidly with the integration of artificial intelligence. i will explore how ai agents, large language models, and automated workflows are transforming the way we build, maintain, and scale data platforms.
writing dbt tests and documentation is often the most neglected part of data engineering. i will show you how i use cursor and ai agents to automate this process, ensuring high-quality data pipelines without the manual overhead.
how i use dbt tests, including not_null, unique, relationships, accepted_values, and singular tests, with examples you can copy to catch data issues early
i walk through how dbt docs work, including what they generate, writing descriptions and doc blocks, persist_docs, and hosting them free on github pages
how snowflake compares to amazon rds and dynamodb, covering architecture, scalability, and when to use each, for data engineers getting started