Summary

Simon Späti argues that data engineering needs a 'grammar' — a declarative language for expressing transformations that can be defined once and executed across multiple compute engines without modification. Drawing on linguistic analogy (sources as nouns, transforms as verbs, templates, modifiers), he explains how Ibis provides the expression layer and xorq extends it with cross-engine execution, content-addressed YAML manifests, lineage tracking, and git-native storage. The core pattern is write → manifest → execute, where the manifest step produces a deterministic, hashed artifact that can run on DuckDB, Postgres, Snowflake, DataFusion, or others interchangeably. Xorq uses Apache Arrow RecordBatch streams as the data pipe between engines, avoiding CSV/JSON serialization overhead. The author positions this 'horizontal data stack' as the antidote to both vertically integrated silos and the inner-platform effect, and argues the same grammar benefits LLMs by giving them structured, reproducible, interchangeable artifacts to reason about.

Key Insight

A declarative grammar for data engineering — where transformations are expressed once as composable, engine-agnostic expressions serialized into content-addressed manifests — eliminates the anti-pattern of re-implementing logic per system and makes pipelines reproducible by both humans and AI agents across any execution engine.

Spicy Quotes (click to share)

• 3

  Having a grammar for data engineering means we can express the workloads in a declarative manner, and then be sure we can deterministically reproduce and apply that exact definition.

• 3

  The moment a verb is applied, the expression stops being a plain noun and becomes a statement, a description of 'data plus what should happen to it.' But the sentence isn't spoken yet, it stays inert, fully composed but unexecuted, until something finally asks it to run.

• 4

  Writing the sentence and saying it out loud are two different acts.

• 5

  unix : programs :: xorq : arrow-transforms

• 4

  UDA is a semantic layer defining what data means across systems. Xorq is a computational layer defining what transformations do across engines. Both reject the same anti-pattern of re-implementing the same logic for every system.

• 3

  Expressions are tools, Arrow is the pipe.

• 3

  With just these four — noun, verb, template, modifier — you can read (and write) arbitrarily complex data pipelines the same way learning a handful of verb-and-object combinations in a text editor lets you compose arbitrarily complex edits.

• 2

  It's all about having high-quality context in the right format, with a clear definition where humans and AI agents can interchange and help each other.

Tone

technical, optimistic, explanatory