Summary

This post is Part 2 of a series demonstrating the 'Grammar of Data' concept applied to a real data engineering digest project that tracks momentum of DE tools using RSS feeds, Bluesky firehose, PyPI downloads, and GitHub events. The author builds the stack with dlt for ingestion, Ibis expressions for transformations, and xorq for engine-agnostic execution across DataFusion, DuckDB, and Snowflake. The key architectural insight is that model and execution are separated: a single stack.yaml swap changes the engine without touching any transformation code. The digest reveals that query/dataframe engines (ibis +49%, sqlglot +45%, DuckDB +42%, Polars +36%) are the fastest-growing DE packages, DuckDB dominates Bluesky buzz, and pydantic leads in raw download volume. The post concludes that the grammar guarantees faithfulness and reproducibility but not correctness—correctness requires reviewed semantic definitions, explored in the next part.

Key Insight

Separating transformation logic from execution engine through a declarative grammar lets teams write metrics once and run them anywhere—from laptop to warehouse—while a manifest system surfaces engine-specific semantic differences before any data is touched.

Spicy Quotes (click to share)

• 4

  Model once, represent everywhere - the transformation never changes, only the engine binding does.

• 4

  The manifest captures that difference explicitly even before we run anything, reviewable instead of a silent runtime error.

• 5

  Because the grammar separates *what* from *where*, you don't rewrite the metric. You just point the noun at the warehouse and run the same sentence.

• 2

  Blog coverage lags the download signal, which is precisely why the digest triangulates four sources instead of trusting just one.

• 4

  The grammar buys us two guarantees: answers that are faithful to the expression that produced them, and reproducible whenever we rerun it. But that doesn't always mean correct.

• 3

  DuckDB is the tool that is both growing fast and the 'loudest'.

• 3

  Deploying a model collapses into the same write → manifest → execute cycle as deploying a metric.

• 2

  xorq gives us the bottom-up approach, working with our data, mapping all parts of the DE lifecycle, running it locally on multiple engines and discovering errors at pre-run time.

Tone

technical, demonstrative, practical