Summary

Späti argues that operationalizing data orchestration requires treating it with the same rigor as production OLTP systems, since the entire data stack depends on it. The post positions GitOps as the foundational practice for managing orchestration deployments, separating infrastructure code from business pipeline logic. The central architectural insight is Dagster's 'code locations' concept, which creates a natural boundary between the data platform team and domain-specific data practitioners. This separation enables teams to deploy, test, and iterate on pipelines independently while sharing a central orchestration platform. The post closes by framing orchestration as the 'operating system' of the modern data platform, increasingly critical as AI workloads amplify data engineering demands.

Key Insight

Treating data orchestration as production-grade infrastructure — with GitOps, separated code locations, and embedded governance — is what transforms a data platform from a fragile collection of pipelines into a composable, enterprise-ready operating system.

Spicy Quotes (click to share)

• 4

  Operating the data orchestration layer can be as critical as the source OLTP databases, when decisions are taken from the data assets produced by downstream data pipelines.

• 8

  DevOps is what data engineering was a decade ago, when everyone was talking about data science, but the actual work was 80% data engineering.

• 2

  Code locations are key to a successful orchestration deployment story where we differentiate business from technical implementation logic.

• 4

  A deployment is only as trustworthy as the data it produces.

• 3

  The data pipeline is exactly where the business rules live, so it's where domain experts can enforce checks and data types they know must hold.

• 3

  Exchanging compute engines and using the best tools for the task is just one config change away, and doesn't need to change the logic of the data pipeline itself.

• 2

  With workspaces you can containerize your data pipeline logic, with its required Python libraries, into a docker image or similar, and test it and deploy independently from the deployment code.

Tone

technical, prescriptive, practitioner-focused