Summary

Simon Späti reflects on his 20+ year career in data engineering, tracing the field's evolution from Business Intelligence in the 1980s through Big Data to today's AI-assisted workflows. He argues that despite constant tool changes, data engineers solve the same fundamental problems: ingesting, modeling, transforming, and serving data. The post emphasizes that mastering fundamentals—data modeling, SQL, and understanding business needs—matters far more than chasing new frameworks. Späti positions data engineers as 'invisible plumbers' whose work goes unnoticed until something breaks, and advocates for embracing this foundational role rather than seeking credit.

Key Insight

Despite 50 years of tool evolution from star schemas to AI agents, data engineering fundamentals remain unchanged—and the engineers who master business understanding and data modeling will always be more valuable than those who chase the latest frameworks.

Spicy Quotes (click to share)

• 6

  You ingest data. You model it. You transform it. You serve it. Someone asks for a change. Everything breaks. You rebuild. This is the loop.

• 7

  The tools change. The loop doesn't.

• 7

  When you do your job, you're invisible. When anything goes wrong, you're under a microscope.

• 8

  Excel isn't the enemy. Excel is the business telling you what they actually need.

• 8

  We're not really any smarter than the people before us. We just have better marketing.

• 7

  Schema changes are usually people problems, not technology problems.

• 4

  The best data engineers aren't the ones who know every new tool. They're the ones who know why the data matters.

• 5

  That's the job. Not the tools. Not the frameworks. Not the buzzwords. The moment when data helps a human make a better decision.

Tone

reflective, pragmatic, confessional