Summary

This interview with Josh Wills — 25-year data engineering veteran and creator of dbt-duckdb — frames data engineering as fundamentally the cache invalidation (materialized view) problem: deciding what to pre-compute now versus compute on demand. Wills argues this core problem won't be automated away by AI agents, primarily because large-scale Spark and Ray pipelines lack the fast feedback loops that make agents effective in domains like SQL, math, and frontend. He describes his current role as an 'emergency exception handler' — a pilot on autopilot who takes over when agents fail — while delegating the tedious work (flaky tests, DB migrations) to agents entirely. The post also covers his view that model choice is more about personal preference than capability, and that what remains irreducibly human in engineering is making and communicating architectural trade-offs.

Key Insight

Data engineering is the cache invalidation problem in batch-processing form, and until AI agents can operate with fast feedback loops at trillion-token pipeline scale, experienced engineers remain essential — not as coders but as emergency exception handlers who make and communicate the trade-offs agents cannot.

Spicy Quotes (click to share)

• 7

  It's the two hard problems in computer science, right? Cache invalidation and naming things. And the agents still suck at them because they're legit hard.

• 8

  You can easily vibe code a job in Databricks that costs like two hundred thousand dollars a day to run, that a competent data engineer, or at least one who supervised it, would never have written.

• 3

  What's tricky about data pipelines, at least at a large scale, is that you don't actually have that fast feedback mechanism available to you, because running a giant pipeline can take a long time and a lot of computers.

• 6

  I have an agent whose whole job is just to watch for flaky tests and fix them. That's it, that's all it does. Database migrations? Absolutely not. I have an agent that does database migrations.

• 5

  It's really the decisions we need to communicate. The decisions build the shared understanding, the choices we make, the trade-offs. That's still the meat of engineering work.

• 7

  Through years and years of suffering and writing gigantic pipelines, I can look at a pipeline and it kind of feels like something to me. I can feel the tuples flowing through it, because I have a mental model of how this is going to go.

• 9

  the fact that Opus 5 is so bad, it deeply indicates to me that we literally have no idea what we're doing at all when it comes to training these models.

• 4

  I can dial up and down the work on demand as I see fit. If I'm in a moment where I need five agents going, I can do it... I can maintain a certain pace to match my energy, to match what I'm up to right now.

Tone

conversational, candid, technically grounded with philosophical tangents