Summary

Simon Späti interviews Chris Riccomini on AI's impact on data engineering, covering correctness in financial data, the case for LLMs speaking Substrait instead of SQL, security concerns with AI agents, and the future of the data engineer role. Chris argues that AI will handle the majority of data engineering work once tooling catches up, and that the distinct 'data engineer' role may fold into a unified data role. The conversation emphasizes plan-mode-first workflows, quality gates, and managing LLM context through techniques like the Ralph Loop. Chris also suggests that programming language choice will increasingly be driven by agent ergonomics—performance, stability, and token cost—rather than human preference.

Key Insight

AI is already capable of handling most data engineering work, but the tooling, security models, and organizational practices haven't caught up—and when they do, the distinct data engineer role may not survive as a separate discipline.

Spicy Quotes (click to share)

• 4

  I think this is a reasonable fear, but it's also a problem we had before AI.

• 3

  You need to have the LLM iterate on the plan for many iterations. Probe its plan, ask it for details, ask it to expand sections.

• 7

  I'll be frank: I think AI will do the majority of the data engineering work in the future. I think we're already at a point where it can; the tooling and practices just haven't yet adapted.

• 6

  We over-specialized the data space. It might have been necessary, but it isn't now.

• 7

  In that world, I just don't care about the language my software is written in.

• 5

  I care more about the characteristics of the output: its performance, stability, and cost to build (i.e. tokens).

• 8

  I think I would have learned a lot more without AI. But I also wouldn't have done the work.

• 5

  I was very disappointed in their (lack of) security model. It's the most important part, and it was completely lacking.

Tone

practical, forward-looking, conversational