Simon Späti

Key Insight: Creation's true value lies in the process itself — the flow state, emotional resolution, and deep happiness it generates — not in external validation or measurable outcomes.

Späti reflects on creation as both an intrinsic act and a gift to the world, arguing that the process matters more than the outcome. Drawing on Picasso, the 'Pathless Path,' and the Artist's Way, he frames creation as following one's subconscious without boundaries to reach a deep flow state. He redefines success through a scientist's lens: success is the experience and learning, not a predetermined result. Creation is presented as an antidote to shallow, passive consumption, yielding deep and lasting happiness. The post concludes that sharing creation as a gift — with no strings attached — frees the creator from the burden of pleasing others.

Key Insight: A declarative grammar for data engineering — where transformations are expressed once as composable, engine-agnostic expressions serialized into content-addressed manifests — eliminates the anti-pattern of re-implementing logic per system and makes pipelines reproducible by both humans and AI agents across any execution engine.

Simon Späti argues that data engineering needs a 'grammar' — a declarative language for expressing transformations that can be defined once and executed across multiple compute engines without modification. Drawing on linguistic analogy (sources as nouns, transforms as verbs, templates, modifiers), he explains how Ibis provides the expression layer and xorq extends it with cross-engine execution, content-addressed YAML manifests, lineage tracking, and git-native storage. The core pattern is write → manifest → execute, where the manifest step produces a deterministic, hashed artifact that can run on DuckDB, Postgres, Snowflake, DataFusion, or others interchangeably. Xorq uses Apache Arrow RecordBatch streams as the data pipe between engines, avoiding CSV/JSON serialization overhead. The author positions this 'horizontal data stack' as the antidote to both vertically integrated silos and the inner-platform effect, and argues the same grammar benefits LLMs by giving them structured, reproducible, interchangeable artifacts to reason about.

Key Insight: AI agents become trustworthy in data engineering only when the probabilistic LLM layer is separated from factual operations — SQL parsing, validation, lineage — which must run on a deterministic core that proves correctness rather than guessing it.

Simon Späti argues that AI agents are most valuable in data engineering when built on a deterministic correctness layer, not used as general-purpose chat tools. He outlines three levels of AI agent maturity — chat-phase, autonomous CLI-access, and dedicated tooling — with dedicated tooling being the most reliable because it offloads factual operations to deterministic engines. The core risk he identifies is not errors that crash pipelines, but silent wrong numbers: queries that compile and pass tests while introducing incorrect joins or inflated metrics. He uses Altimate Code as a concrete example of the architecture he advocates: a probabilistic LLM layer on top of a deterministic Rust/TypeScript core that handles SQL parsing, validation, and equivalence checks sub-millisecond. The post includes a practical blast-radius analysis example showing how such tooling can automatically map downstream impact before a column rename. His conclusion is that general-purpose agents are fine for exploration, but any output touching production requires a deterministic core underneath.

Key Insight: A note-taking system earns its worth not by how thoroughly it captures information, but by how frictionlessly it converts private thinking into public sharing — and that requires owning your format and collapsing the distance between note and publication.

Simon Späti reflects on 17 years of note-taking evolution, sharing the system and principles that now power his public writing and livelihood. The core insight he's internalized is that writing is not the output of thinking but the medium through which thinking happens. He advocates for connected notes (Zettelkasten) over folder hierarchies, open plain-text formats over proprietary apps, and minimal friction workflows that encourage sharing. His current stack — Obsidian, Markdown, GoHugo, and Quartz — collapses the gap between private notes and published content to near zero, requiring only a #publish tag and a script. The post is a personal love letter to the practice, arguing that the best note system is the one you actually enjoy using.

Key Insight: Treating data orchestration as production-grade infrastructure — with GitOps, separated code locations, and embedded governance — is what transforms a data platform from a fragile collection of pipelines into a composable, enterprise-ready operating system.

Späti argues that operationalizing data orchestration requires treating it with the same rigor as production OLTP systems, since the entire data stack depends on it. The post positions GitOps as the foundational practice for managing orchestration deployments, separating infrastructure code from business pipeline logic. The central architectural insight is Dagster's 'code locations' concept, which creates a natural boundary between the data platform team and domain-specific data practitioners. This separation enables teams to deploy, test, and iterate on pipelines independently while sharing a central orchestration platform. The post closes by framing orchestration as the 'operating system' of the modern data platform, increasingly critical as AI workloads amplify data engineering demands.

Key Insight: Reliable analytics from AI agents depends less on the model and more on a governed, git-reviewed context layer that unifies hard warehouse semantics with soft business knowledge, giving agents a warm start that is both more accurate and cheaper.

Späti argues that pointing an AI agent like Claude or Codex at a data warehouse produces mediocre results because the agent lacks knowledge of joins, metric definitions, and business rules that give numbers meaning. He proposes a 'context layer' as an evolution beyond the semantic layer—one that captures both 'hard semantics' (warehouse schema, joins, and metric definitions as YAML and SQL) and 'soft semantics' (business knowledge living in docs, wikis, and Notion). Both are committed to git and reviewed like code, keeping humans in the loop while giving agents a 'warm start' instead of a cold database connection. The bulk of the post introduces ktx, a new open-source context layer from Kaelio, walking through its CLI commands for discovering metrics, validating SQL, searching the wiki, and serving governed context to agents via MCP. The payoff Späti emphasizes is accuracy plus literal cost savings, since a well-contextualized agent makes fewer and cheaper warehouse queries. The piece is openly a sponsored product explainer written as part of his consulting services.

Key Insight: In an era where code is essentially free to generate, the engineer's value shifts from writing code to exercising taste, judgment, and disciplined process—deciding what to build and rigorously reviewing what agents produce, which is what separates responsible agentic engineering from dangerous vibe coding.

In the third installment of his 'How to use AI with DE' interview series, Simon Späti talks with Wes McKinney about how he actually works with coding agents. The central distinction is that what Wes practices is 'agentic engineering'—rigorous spec-driven, reviewed, and tested work—not 'vibe coding,' which he calls dangerous and irresponsible. Wes invests heavily in the specification phase, uses the superpowers framework to prevent agent drift, and auto-reviews every commit with his own tool Roborev, leaving him with far less but higher-quality code to review himself. Because code is now cheap to create but expensive to maintain, he argues that saying no and exercising taste is the engineer's last line of defense against AI-generated bloat. He believes paying the true cost of tokens will eventually curb wasteful, low-value work, and that the future lies in knowledge systems, semantic layers, and headless BI rather than traditional data engineering. The lingering open question is how the next generation of senior engineers will develop expertise when the hard, hands-on coding labor—where people normally learn—is being automated away.

Key Insight: Dagster's evolution from ETL orchestrator to open data platform demonstrates that composability and declarative asset-based thinking are the keys to taming enterprise data complexity.

Simon Späti presents a comprehensive 'almanack' of Dagster, tracing its evolution from a developer-friendly ETL framework in 2018 to a full open data platform. He argues that Dagster's key innovations — data-aware orchestration, the shift from task-based DAGs to declarative data assets, and composable architecture — address the inherent complexity of enterprise data systems. Drawing on Rich Hickey's 'Simple Made Easy' philosophy, he frames composability as the antidote to enterprise data complexity. The post positions Dagster as the right abstraction layer for building open data platforms, with its control plane unifying metadata, lineage, and observability across heterogeneous systems.

Key Insight: AI is already capable of handling most data engineering work, but the tooling, security models, and organizational practices haven't caught up—and when they do, the distinct data engineer role may not survive as a separate discipline.

Simon Späti interviews Chris Riccomini on AI's impact on data engineering, covering correctness in financial data, the case for LLMs speaking Substrait instead of SQL, security concerns with AI agents, and the future of the data engineer role. Chris argues that AI will handle the majority of data engineering work once tooling catches up, and that the distinct 'data engineer' role may fold into a unified data role. The conversation emphasizes plan-mode-first workflows, quality gates, and managing LLM context through techniques like the Ralph Loop. Chris also suggests that programming language choice will increasingly be driven by agent ergonomics—performance, stability, and token cost—rather than human preference.

Key Insight: External tables represent one of data engineering's most durable patterns because the principle of reading data in place rather than moving it only grows more valuable as formats evolve, and with modern lakehouse architectures the line between external and internal storage is finally dissolving.

Simon Späti traces the 25-year history of external tables from Oracle 9i in 2001 to modern lakehouse architectures, arguing they represent one of data engineering's most enduring patterns. He explains that external tables are essentially pointers to data stored outside a database, queryable via SQL without moving data. The post compares internal vs. external storage tradeoffs around query speed, cost, and data freshness, with benchmarks showing external tables pay only a 1.3×–1.7× performance tax while offering up to 20× storage cost savings. Späti shows how the pattern evolved from CSV parsing to open table formats with ACID guarantees, and argues that with DuckLake and open catalogs, the distinction between external and internal is dissolving. He invokes the Lindy Effect to predict external tables will persist another 25 years.

Key Insight: AI agents don't eliminate the need for BI — they expose that the real value was always in the governed metrics, semantic layers, and maintained infrastructure underneath the dashboards.

Simon Späti argues that BI was never really about dashboards — it was about the primitives underneath: metrics, semantics, governance, and trusted definitions. He traces the recurring 'BI is dead' narrative through industry voices like Benn Stancil, Hex, and Rill's Mike Driscoll, showing that each wave rediscovers the same truth. While AI agents can generate dashboards and chat-based analytics, they actually make BI's foundational layers more important, not less, because agents hallucinate without governed semantic layers. The post highlights maintenance as the elephant in the room: generating dashboards is easy, but maintaining them at scale is where real value lies. Späti proposes BI-as-Code as a path toward maintainable, versioned, agent-friendly BI infrastructure. The conclusion is that BI primitives are not casualties of the AI era but essential infrastructure for it.

Key Insight: Consistent AI results come not from better prompting but from rigorous upfront specification — spending hours on specs and treating initial builds as disposable explorations produces far better outcomes than iterating on generated code.

Simon Späti interviews Mark Freeman about his Spec-Driven Development (SDD) workflow for producing consistent, high-quality results with AI coding agents like Claude Code. Mark's approach centers on spending extensive time defining requirements through ExcaliDraw diagrams, JSON schemas, and markdown specs before letting agents build, then assessing outcomes against specs rather than reviewing code directly. The first build is deliberately treated as throwaway — a form of requirements exploration — with learnings fed back into updated specs for subsequent iterations. Mark argues AI agents benefit senior engineers far more than juniors, since experience is needed to make sound architectural decisions and avoid accumulating early legacy code. The interview also covers agent parallelization with tmux and Agent Teams, the role of evals in data contract work, and the addictive 'Claude Code slot machine' dynamic of shipping AI-generated code without learning.

Key Insight: The most agent-friendly analytics architecture turns out to be the one built on old principles — local-first, text-based, SQL-defined, and version-controlled — because tools designed for developer simplicity naturally provide the readable context that AI agents need.

Simon Späti argues that local-first, developer-friendly tools like MotherDuck (serverless DuckDB) and Rill (BI-as-code) are naturally suited for AI agent workflows because what's readable by developers is also readable by agents. He demonstrates how declarative YAML dashboards, SQL-based metrics layers, and CLI-first workflows create an architecture where agents can read, reason about, and build analytics autonomously. The post walks through three practical examples including Stack Overflow survey analytics and multi-cloud cost analysis. Späti contends that dashboards won't disappear but will coexist with conversational BI, and that the key enabler is having semantic context defined as code. He acknowledges limitations around natural language ambiguity and the tension between AI's non-determinism and data's need for reproducibility.

Key Insight: The future of blogging is not standalone posts but a connected web of living notes and frozen articles that compound knowledge the way the brain naturally learns—and human authorship matters more than ever precisely because AI has made authentic voice a scarce resource.

Simon Späti reflects on a decade of blogging, arguing that personal writing remains valuable despite AI and social media disruption. He advocates for a 'connected' future of blogging through linked second brain notes that mirror how the human brain actually learns. His core thesis is that blogs and notes serve complementary purposes: blogs capture moments in time while notes compound and evolve. He sees manual, genuine writing as increasingly important—not less—in an era of AI-generated content. The post doubles as a detailed breakdown of his personal workflow using Obsidian, Markdown, and Vim motions.

Key Insight: Every mature Git-for-data tool converges on the same core trick—separating metadata from data via pointer manipulation—but they diverge significantly on merge support, granularity, and infrastructure fit, so choosing the right tool requires matching those trade-offs to your specific stack and workflow.

This is Part 2 of a series on Git-for-data workflows, examining how tools like LakeFS, Dolt, Nessie, MotherDuck, Bauplan, Neon, and DuckLake implement version control semantics for data without copying petabytes. The central insight is that all mature tools converge on the same architectural principle: separating metadata from data, using copy-on-write and pointer manipulation to enable instant branching. The post breaks tools into three categories—data lake versioning, transactional databases, and analytical warehouses—each with different trade-offs around merge support, branching granularity, and infrastructure requirements. Beyond storage, the post extends the analysis to orchestration (Dagster branch deployments) and AI agent workflows, showing Git-like patterns spreading across the full data engineering lifecycle. The author concludes that Git-like workflows are becoming table stakes, and recommends starting with high-risk pipelines before expanding.

Key Insight: AI agents accelerate building but the real productivity unlock is plan mode — structured human-in-the-loop planning before autonomous execution — not smarter models.

Simon Späti built a personal knowledge assistant using RAG on his Obsidian vault, with DuckDB as the vector database locally and MotherDuck for a serverless web app via WASM. The system uses BGE-M3 embeddings to enable semantic search across nearly 9,000 markdown notes, leveraging Obsidian's backlink graph to boost results and surface hidden connections between notes. He emphasizes the local-first approach to keep sensitive notes private. The article doubles as a reflection on AI agents in data engineering, arguing that 'plan mode' in tools like Claude Code is the real productivity breakthrough — not smarter models. He cautions that vibe coding and AI agents still require a human architect to avoid generating unmaintainable code that solves the wrong problem.

Key Insight: Switching from macOS to Linux is now viable for power users thanks to Omarchy's polish and AI-powered troubleshooting with Claude Code, trading Apple's seamless defaults for deeper customizability and permanent fixes.

Simon Späti shares his 8-month experience after switching from macOS to Arch Linux with Omarchy as his daily driver. He details the Linux replacements for his macOS apps including Walker Launcher for Raycast, Morgen for calendar, and Filen for backups, finding many alternatives equal or superior. He documents running Windows inside Linux via Docker for Microsoft Office, and upgrading to a Tuxedo InfinityBook Pro 14 with 128GB RAM. While praising the terminal-native workflow and customizability, he honestly catalogs significant pain points including GPU crashes, hibernation bugs, and WiFi firmware issues. He concludes that Claude Code was essential for troubleshooting and that Linux rewards tinkering with deeper understanding and lasting fixes.

Key Insight: StarRocks' colocated joins and multi-tier caching let teams skip the expensive pre-denormalization step in Flink/Spark, but good data modeling and careful partition key planning remain essential for realizing those gains.

This sponsored deep-dive examines why companies like Coinbase, Pinterest, and Fresha chose StarRocks over alternatives like ClickHouse, Druid, and Pinot for their analytics needs. Through interviews with engineers at these companies, Späti identifies a common pattern: customer-facing analytics on cloud data warehouses like Snowflake became too slow, and teams needed sub-second query responses without heavy pre-denormalization. StarRocks' key differentiator is its ability to perform fast distributed joins natively, enabled by colocated joins, a cost-based optimizer, and multi-tier caching. The article concludes that StarRocks excels when joins are central to your analytics, while ClickHouse remains stronger for single-table observability workloads. Despite the advantages, good data modeling is still essential, and the smaller community and operational complexity are real tradeoffs.

Key Insight: Despite 50 years of tool evolution from star schemas to AI agents, data engineering fundamentals remain unchanged—and the engineers who master business understanding and data modeling will always be more valuable than those who chase the latest frameworks.

Simon Späti reflects on his 20+ year career in data engineering, tracing the field's evolution from Business Intelligence in the 1980s through Big Data to today's AI-assisted workflows. He argues that despite constant tool changes, data engineers solve the same fundamental problems: ingesting, modeling, transforming, and serving data. The post emphasizes that mastering fundamentals—data modeling, SQL, and understanding business needs—matters far more than chasing new frameworks. Späti positions data engineers as 'invisible plumbers' whose work goes unnoticed until something breaks, and advocates for embracing this foundational role rather than seeking credit.

Key Insight: Escaping algorithm-dominated platforms for open protocols and personal blogs isn't just about technology choice - it's essential for reclaiming the foundational pillars of well-being: health, family connection, and freedom from status-seeking greed.

Simon Späti argues that algorithm-dominated platforms are destroying our well-being by constantly grabbing attention and preventing genuine human connection. He identifies three foundational pillars for a good life: health, family/community, and self-contentment (avoiding greed). The essay critiques how social media addiction leads to shallow happiness and broken relationships, while advocating for a return to the open web through protocols like AT Protocol and Bluesky. Späti sees personal blogs and direct follower connections as the antidote to algorithmic control, predicting that as people tire of AI-generated content and platform dependence, they'll return to authentic human-created content and real connections.

Key Insight: The choice between open-source flexibility and opinionated platforms is increasingly bridged by open standards, but DevOps complexity makes unified platforms the pragmatic choice for teams prioritizing time-to-value over customization.

Simon Späti argues that the data platform landscape is shifting from fragmented 'Modern Data Stack' approaches toward unified, opinionated platforms that handle infrastructure decisions for you. Using the Japanese concept of 'Omakase' (chef's choice) as a metaphor, he contrasts the flexibility of open-source tool assemblies against the operational simplicity of integrated platforms. He emphasizes that open standards (like Parquet, Iceberg, Arrow) are the crucial bridge enabling both approaches to coexist and data to remain portable. The post concludes that DevOps complexity is the biggest challenge in data engineering today, and opinionated platforms solve this by providing battle-tested infrastructure, native observability, and separation of concerns out of the box.

Key Insight: DuckDB's lightweight caching extensions offer a pragmatic middle ground between slow uncached queries and the operational complexity of traditional OLAP systems, letting the database itself serve as the cache.

The post explores how DuckDB and its extensions offer a simpler alternative to traditional OLAP cubes for caching analytical workloads. Simon traces the history of BI caching from data warehouses through OLAP cubes to modern zero-copy approaches, arguing that DuckDB's lightweight binary and transparent caching extensions like QuackStore and cache_httpfs can dramatically speed up dashboards without requiring separate ingestion pipelines. He demonstrates practical examples showing 10-15x query speedups and introduces the concept of 'Frozen DuckLake' for immutable reference data. The core thesis is that the database itself can now serve as the cache, eliminating traditional cache invalidation complexity.

Key Insight: A declarative, code-first data stack with tools like dlt and Rill provides the flexibility to easily switch between local and cloud deployments while keeping full control over configuration and automation.

This tutorial extends a local FinOps project to cloud-native infrastructure using ClickHouse Cloud and Rill Cloud for multi-cloud cost analytics. Building on a previous local Parquet-based solution, Späti demonstrates how dlt's destination flexibility makes switching from local files to ClickHouse trivial with just an environment variable change. The article walks through ingesting AWS CUR, GCP cost, and Stripe data, deploying interactive dashboards to Rill Cloud, and automating the pipeline with GitHub Actions. The hardest challenges weren't the tools themselves but the coordination work: SQL dialect differences between DuckDB and ClickHouse, hard-coded table names, and data anonymization requirements.

Key Insight: Git-like workflows for data require metadata-based zero-copy cloning rather than physical data duplication, enabling instant branching of production datasets for testing without the time and cost of traditional environment setup.

The post explores applying Git-like version control workflows to data management, addressing the challenge of testing and rolling back data pipelines in production environments. Simon argues that while Git revolutionized code versioning, data presents unique constraints: tables must remain consistent across relationships, production datasets are massive, and copying data for testing is slow and expensive. The solution lies in leveraging metadata catalogs and zero-copy cloning through technologies like Prolly Trees, which enable instant branching of production data without physical duplication. The article establishes a data movement efficiency spectrum from metadata-based versioning (most efficient) to full data copying (least efficient), setting up a Part 2 that will examine specific tools implementing these concepts.

Key Insight: Building multi-cloud cost visibility requires navigating each provider's unique export mechanisms before any real analytics work can begin, but a composable stack of dlt, DuckDB, and Rill makes the unified dashboard achievable and extensible.

Simon Späti presents a comprehensive tutorial for building a multi-cloud cost analytics dashboard that unifies AWS, GCP, and Stripe revenue data into a single view. The project uses dlt for data extraction, DuckDB/Parquet for storage, and Rill for visualization. He documents the unexpected complexity of cloud cost exports—AWS requires S3 bucket setup, GCP needs BigQuery configuration—before data can even be extracted. The end result is a composable, code-first solution that calculates margins by combining cloud costs with revenue, giving businesses visibility they typically lack. He also demonstrates how AI coding agents excel with declarative data stacks where everything is local and code-defined.

Key Insight: The backlash against algorithmic control is driving people toward single-purpose tools, owned media, and intentional boredom as antidotes to attention exploitation.

Simon argues that we're at a cultural turning point where people are rejecting algorithm-driven experiences in favor of owning things, using single-purpose devices, and embracing distraction-free tools. He advocates for boredom as a luxury that heals mental health and sparks creativity, contrasting the shallow consumption of algorithmic feeds with the deep, transformative experience of reading books. The post weaves together observations about technology backlash with personal book recommendations and reflections on writing from a blank page. He concludes that the pendulum is swinging back toward intentional, human-centered experiences over corporate-optimized ones.

Key Insight: Success in data engineering comes from understanding business context and stakeholder needs, not from mastering the trendiest tools—fundamentals, communication, and pragmatic simplicity consistently outweigh technical complexity.

This article presents a roundtable discussion where four senior data engineers (Ben Rogojan, Julien Hurault, Mehdi Ouazza, and Simon Späti) answer the top 10 most-upvoted questions from r/dataengineering. The questions span interview preparation, data quality, warehouse vs lakehouse decisions, schema changes, real-time requirements, inheriting legacy systems, Linux skills, Excel exports, cloud costs, and career wisdom. The experts consistently emphasize fundamentals over trendy tools, understanding business context, starting simple and iterating, and the importance of communication with stakeholders. The format draws from Reddit's 174K-member community to surface the real challenges data engineers face daily.

Key Insight: Effective agentic analytics requires structured semantic layers that give AI precise business context, sub-second speed that lets humans verify results, and explicit stewardship guardrails — treating agents as advanced linters rather than autonomous replacements.

Späti argues that AI agents in analytics need three pillars to work effectively: semantics (structured metric definitions), speed (sub-second query responses), and stewardship (guardrails and human oversight). He proposes Metrics SQL as a higher-level SQL dialect that operates on business concepts rather than raw columns, making it ideal for both human and AI consumption. The article demonstrates how tools like Rill with MCP connections enable agentic workflows where AI can query and create metrics while respecting defined constraints. His core thesis is that data modeling fundamentals haven't changed, but agents now serve as powerful collaborators — more like advanced linters than replacements for human judgment.

Key Insight: Declarative YAML-based configurations can replace most custom ETL code, offering enterprise features like SCD2, cross-engine SQL transpilation, and automatic DAG generation without vendor lock-in.

This sponsored article explores how declarative data stacks, specifically Starlake, can replace custom ETL code with YAML configurations. Späti argues that most data teams waste 80% of their time on infrastructure plumbing—writing UPSERT logic, building incremental loads, and debugging orchestration failures. He walks through Starlake's capabilities across the full ETL lifecycle: extraction from diverse sources, loading with built-in write strategies like SCD2 and UPSERT, SQL transformation with cross-engine transpilation, automated DAG generation, and testing locally with DuckDB. The core pitch is that a configuration-first approach eliminates glue code and vendor lock-in while delivering enterprise features that normally require dedicated platform teams.

Key Insight: The most authentic and inspiring technical writing comes from lived experience rather than desk research, because a writer's goal should be to spark new thinking, not to relay information.

Simon Späti explains why he deliberately avoids traditional research when writing, preferring to draw entirely from personal experience and accumulated knowledge. He views his ongoing note-taking in his 'Second Brain' as a form of passive research that connects ideas over time. With two decades of field experience, he argues that things don't change as drastically as media suggests. His goal as a writer is not to recall information but to spark thinking and inspire readers through authentic, experience-based perspectives.

Key Insight: ClickHouse can serve as both your transformation engine and storage layer, handling complex aggregations at ingestion time rather than requiring traditional ETL pipelines, while its modeling strategies let you explicitly choose where in the pipeline to handle complexity based on your specific performance and freshness requirements.

This comprehensive guide explains how to model data effectively in ClickHouse for real-time analytics, moving from theory to a practical implementation using NOAA weather data. The author argues that ClickHouse can serve as both storage and transformation engine, eliminating traditional ETL bottlenecks. Key strategies covered include denormalization, materialized views, partitioning, and pre-aggregation using MergeTree engines. The post demonstrates a complete data pipeline from S3 ingestion through ClickHouse transformation to Rill visualization, all configured in a single YAML file. The fundamental principle is minimizing joins at query time while choosing where in the pipeline to handle complexity based on performance and freshness requirements.

Key Insight: Semantic layers matter most when you need to maintain consistent business metrics across multiple analytics tools, enabling ad-hoc queries with dynamic granularity while keeping metric definitions in a single, version-controlled location.

The post argues that semantic layers provide crucial value by defining business metrics once in a single source of truth, eliminating duplication across BI tools, web apps, and notebooks. Simon demonstrates this practically by building a minimal semantic layer using YAML definitions and Python/Ibis with DuckDB as the query engine, analyzing 20 million NYC taxi records. He emphasizes that semantic layers are most valuable when you have multiple analytics consumers, complex business logic, or need ad-hoc queries with dynamic granularity. The article distinguishes between physical datasets and logical aggregations, showing how the abstraction layer enables non-technical users to modify metrics without touching code.

Key Insight: The freedom to fully customize your operating system and workflow can outweigh the polish and convenience of macOS, especially for developers frustrated by Apple's increasingly restrictive approach.

Simon Späti documents his transition from a 5-year-old MacBook Pro M1 Max to a Lenovo ThinkBook running Arch Linux with DHH's Omarchy distribution. After 15 years on macOS, he finds the switch surprisingly smooth thanks to Omarchy's opinionated setup that mirrors keyboard-driven workflows. While acknowledging trade-offs like worse battery life, fan noise, and missing tools like Snagit, he emphasizes the joy of customization and learning. The post details his pre-switch requirements, app replacements found for most needs, and hardware choice rationale, ultimately concluding that the freedom to modify his OS outweighs the polish of macOS.

Key Insight: Mastering data engineering requires consistent, structured learning across SQL, Python, cloud platforms, and DevOps—with SQL remaining the irreplaceable foundation because it enables precision that natural language cannot match.

Simon Späti presents a comprehensive three-week learning roadmap for aspiring data engineers to master essential skills during summer. Week 1 covers foundations (SQL, Git, Linux), Week 2 focuses on core engineering (Python, cloud platforms, data modeling), and Week 3 addresses advanced topics (streaming, data quality, DevOps). The guide emphasizes learning at your own pace with curated resources, stressing that consistency matters more than speed. Späti argues that while you won't know everything after three weeks, you'll understand the problems and identify potential solutions.

Key Insight: Material and career pursuits provide only shallow happiness; lasting fulfillment comes from deep relationships and surrendering the illusion of control to something greater than oneself.

Simon Späti explores life's fundamental purpose through a deeply personal two-year journey from secular Stoicism toward Christianity. He argues that modern society's abandonment of religion correlates with rising anxiety and depression, while material pursuits and career worship provide only shallow, temporary happiness. Drawing from his wife's faith, diverse philosophical traditions, and biblical study, he concludes that deep relationships—with others and potentially with God—provide life's true meaning. The essay positions Christianity's community, worship, and moral framework as superior to secular alternatives for navigating hardship and finding lasting fulfillment.

Key Insight: Modern data engineering requires mastering a spectrum of DevOps abstraction levels and infrastructure tooling, but as AI handles more coding tasks, soft skills and business understanding become the true differentiators.

This second part of Simon Späti's data engineering toolkit series covers advanced infrastructure concerns: data processing tools, DevOps practices, and the expanding role of infrastructure as code. The post argues that modern data engineers must master not just SQL and Python, but also Kubernetes, Terraform, and GitOps practices to build production-grade platforms. Späti presents a comprehensive catalog of tools across databases, orchestration, BI, containerization, and observability, while noting that AI-enhanced workflows are reshaping pipeline development. The piece concludes that soft skills and business understanding are increasingly crucial as AI reduces pure coding demands.

Key Insight: True self-serve BI becomes possible when LLMs can autonomously query semantic layers through MCP, but effectiveness depends on narrowing analytical context and maintaining human oversight from domain experts.

Simon Späti argues that conversational BI powered by Model Context Protocol (MCP) finally delivers on the two-decade-old promise of self-serve business intelligence. Unlike traditional BI where self-serve meant filtering dashboards, MCP enables LLMs to autonomously query databases, verify facts, and generate BI artifacts through natural language prompts. The post identifies three distinct output types—numerical data retrieval, AI-generated text interpretations, and visual charts—that combine to create a more powerful workflow than either humans or AI alone. Späti emphasizes that narrowing the analytical context makes AI more deterministic and less prone to hallucination, while semantic layers provide the crucial business context LLMs need to understand domain-specific metrics.

Key Insight: YAML-based orchestration creates an ideal collaboration layer between humans and AI models, serving as shared context that enables both parties to iteratively build and maintain complex data pipelines.

This practical guide explores enterprise-level data orchestration using Kestra, dlt, and Snowflake. The post demonstrates how to build production-ready data pipelines with proper separation of concerns, secret management, and multi-tenancy. Späti emphasizes the 'universal orchestrator' concept where visual UI interfaces generate version-controlled YAML code, making pipelines accessible to both engineers and analysts. The article also covers AI integration through MCP (Model Context Protocol), arguing that YAML-based orchestration is uniquely suited for LLM collaboration because the code artifacts serve as shared context between humans and AI models.

Key Insight: Open table formats and lightweight compute engines like DuckDB are democratizing the lakehouse architecture, enabling database-like capabilities on cheap storage while avoiding vendor lock-in through interchangeable, standards-based components.

Simon Späti explores the open lakehouse architecture built on affordable object storage combined with open table formats like Iceberg, Delta Lake, and Hudi. He argues that these formats bring database-like features (ACID transactions, versioning, SQL interfaces) to distributed files without expensive SSD storage. The article demonstrates practical examples using DuckDB and MotherDuck to query Iceberg tables directly from S3, and introduces DuckDB as a lightweight catalog layer using views. Späti positions this as the next evolution of the Databricks Lakehouse concept, emphasizing modularity and vendor independence through interchangeable compute engines and open catalogs like Polaris and Unity Catalog. He concludes that while open data platforms trade some performance for flexibility, they enable truly composable data ecosystems without lock-in.

Key Insight: Self-hosting and tech independence isn't just about avoiding subscription costs—it's about the compounding value of ownership, the joy of building and learning, and freedom from platform dependency.

Simon advocates for tech independence through self-hosting and owning your digital infrastructure, arguing that buying your own domain and hosting your own services compounds over time. He shares his personal journey from vim/Neovim to building a homelab, emphasizing that while self-hosting requires effort, the joy comes from learning, using what you build, and not depending on companies that can kill products or raise prices. The post celebrates open-source software and the community that shares code freely, highlighting tools like Listmonk, GoatCounter, and Quartz that power his online presence. His core message: start with a domain, learn Linux fundamentals, and gradually build independence from big tech platforms.

Key Insight: Declarative data stacks represent a paradigm shift where configuration replaces coding, enabling enterprises to manage growing data complexity while reducing technical debt and democratizing data engineering beyond traditional programmers.

Simon Späti argues that declarative data stacks represent the next evolution in enterprise data engineering, combining open-source flexibility with integrated platform capabilities. He traces the journey from monolithic vendor solutions to fragmented modern data stacks, positioning declarative approaches as the synthesis that offers both freedom and cohesion. The core thesis is that YAML-based configuration can replace most custom coding, enabling teams to deliver production-ready platforms faster while maintaining governance. Using Starlake as his primary example, he demonstrates how declarative stacks cover the entire data lifecycle—extract, load, transform, orchestrate—through configuration rather than code. The approach promises reduced technical debt, better reproducibility, and democratized data engineering accessible to non-programmers.

Key Insight: Open table formats like Iceberg are shifting the data industry from vendor lock-in to a composable architecture where storage becomes commoditized and compute becomes interchangeable, fundamentally changing how organizations own and access their data.

Simon Späti argues that open table formats, particularly Apache Iceberg, are revolutionizing data architecture by enabling database-like features on top of distributed files in object storage. He introduces the 'ICE Stack' (Interoperable, Composable, Efficient) as a framework for understanding managed Iceberg tables, which combine cheap storage with flexible compute options. The article examines how hyperscalers like AWS, Databricks, and Cloudflare are betting heavily on managed Iceberg solutions, evidenced by Databricks' $2B acquisition of Tabular. While celebrating the shift toward open standards and vendor lock-in elimination, Späti also critically examines concerns about proprietary APIs in implementations like AWS S3 Tables, noting the tension between managed convenience and true openness.

Key Insight: Data engineering platforms are fundamentally built on logs rather than debuggers, making efficient log parsing with tools like DuckDB a critical competitive advantage for data teams.

Simon argues that logs are the lifeblood of data platforms and that data engineers spend significant time parsing and analyzing them for debugging, tracing, and performance analysis. Unlike software engineers who can use debuggers, data engineers must 'log their way through problems' because data is constantly changing and flowing from external sources. The post provides a comprehensive tutorial on using DuckDB as a log parser, demonstrating its ability to handle various formats (JSON, CSV, syslog) with zero-copy reading from S3, data warehouses, or HTTPS. For larger datasets that exceed local resources, MotherDuck extends DuckDB's capabilities to cloud scale without syntax changes, enabling collaborative analysis of massive log datasets.

Key Insight: Modern data orchestrators must combine declarative code-first approaches with intuitive UIs to bridge the gap between software engineering best practices and accessibility for business users.

Simon Späti argues that universal data orchestrators are the essential 'heartbeat' of modern data engineering, addressing challenges like complexity management, heterogeneous architectures, data quality, and governance. He traces the evolution from cron jobs to declarative YAML-based orchestrators, emphasizing the convergence of infrastructure-as-code principles with data pipeline management. The post advocates for centralized orchestration over implicit event-driven approaches for complex data workloads, highlighting Kestra's dual interface paradigm (UI + Code) as a bridge between technical and business users. Späti concludes that modern orchestrators must support software engineering best practices while remaining approachable to non-programmers.

Key Insight: Declarative code-first approaches have fundamentally changed what's possible with shifting left—enabling data teams to move business logic upstream simply by copying configuration between tools, catching errors during development rather than in production, and making data quality a collective organizational responsibility rather than an afterthought.

This article explains 'shifting left' in data engineering—moving data quality checks, business logic, and transformations earlier in the data lifecycle rather than handling them downstream in BI tools. Simon traces the concept from software testing origins through DevSecOps to modern data practices. He argues that declarative, code-first approaches have made shifting left more practical by enabling logic to move simply from one config file to another. The key benefits include earlier error detection, reduced compute costs through pre-materialization, centralized business logic, and enabling domain experts to contribute to data quality upstream.

Key Insight: Rather than building parallel infrastructure for AI workloads, data engineers should integrate vector capabilities into their existing data platforms to avoid technical debt and leverage established expertise.

The article distinguishes between vector engines (like DuckDB) that use vectorized execution for analytical processing and vector databases (like Pinecone, Weaviate) designed specifically for AI embedding storage and similarity search. Späti argues strongly against building parallel data infrastructure for AI workloads, advocating instead for integrating vector capabilities into existing data engineering stacks. He examines how traditional databases are adding vector extensions and questions whether dedicated vector databases will remain necessary long-term. The piece concludes by positioning DuckDB as a practical bridge between data engineering and AI workflows, offering vector capabilities without requiring separate infrastructure.

Key Insight: The analytical database landscape has come full circle—modern real-time databases return to OLAP principles of co-located compute and pre-aggregation, but with dynamic flexibility enabled by hardware advances, offering a compelling alternative to cloud data warehouses when sub-second latency matters more than infinite scale.

The post examines when organizations should move beyond Postgres for analytical workloads and explores the landscape of specialized analytical databases. It distinguishes between cloud data warehouses (Snowflake, BigQuery) optimized for scale but with higher latency, and real-time analytical databases (ClickHouse, Pinot, Druid) that co-locate compute and storage for sub-second queries. The author traces the evolution from early OLAP cubes through modern systems, noting that the industry has come 'full circle' in balancing performance with flexibility. The piece provides decision frameworks and migration guidance for choosing between these categories based on query patterns, cost constraints, and latency requirements.

Key Insight: DuckDB democratizes geospatial analytics by eliminating the infrastructure complexity and cost barriers that traditionally required expensive GIS software or complex PostGIS setups.

This comprehensive tutorial introduces geospatial data processing using DuckDB, explaining why specialized spatial capabilities are necessary beyond traditional relational databases. The post covers fundamental GIS concepts including spatial data types, coordinate systems, and spatial indexing (R-trees), demonstrating how spatial queries can reduce computations from 100 billion to 10 million through efficient bounding box filtering. Using the Foursquare dataset as a practical example, Späti shows how to perform spatial analysis like finding chocolate store density across Swiss cities and identifying store clusters. The article positions DuckDB and MotherDuck as accessible alternatives to expensive traditional GIS tools, democratizing geospatial analytics for data engineers.

Key Insight: True productivity and creativity come not from optimizing for external rewards and constant stimulation, but from designing environments and habits that support flow states and embracing the slow, patient process of deep work.

Simon Späti explores how to achieve flow states and deep work in an age of digital distractions, drawing from personal experience as a self-employed author. He argues that tech companies manipulate us like lab animals through dopamine-driven engagement, while artists who focus on process over reward naturally enter flow states. The post offers practical strategies for creating focus-friendly environments, the importance of breaks and rest, and embracing slow productivity. He advocates for 'deep happiness' from mastery and meaningful work over 'shallow happiness' from quick dopamine hits, ultimately connecting flow to living a deeper, more fulfilling life.

Key Insight: Pivot tables endure not despite their constraints but because of them - their rigid structure of rows, columns, filters, and values creates the intuitive simplicity that builds user trust and enables standardization across any tool.

Simon Späti traces the history and enduring relevance of pivot tables, from their origins in 1986 at Lotus through Excel's dominance to their modern renaissance in BI tools. He argues that pivot tables succeed because they align with human cognition through simplicity, intuitive constraints, and instant feedback - functioning as a 'REPL for tables.' The post examines how modern data tools like Rill, Cube, and DuckDB are integrating pivot table functionality, powered by fast OLAP backends. Späti predicts that pivot tables combined with AI (GenBI) will become the primary interface for non-technical users to explore data, especially as they become capable of querying data lakes directly.

Key Insight: Mastering foundational tools like Linux commands, Docker, and simple orchestration provides data engineers with more leverage than chasing the latest complex frameworks—sometimes a well-crafted shell pipeline is all you need.

Simon Späti provides a comprehensive overview of the essential tools and foundational skills every data engineer needs, from operating system choices to Linux commands, Docker containerization, and development environments. He emphasizes that while the toolkit is extensive, engineers should focus on fundamentals and learn incrementally rather than trying to master everything at once. The article covers the practical building blocks: environment variables, virtual machines, CLI tools, simple orchestration with Makefiles and cron, IDEs, notebooks, and version control with git. He concludes that SQL and Python remain the core languages, with the reminder that sometimes the simplest Linux command can replace a complex toolchain.

Key Insight: A truly declarative data stack requires treating the entire pipeline as a compiler problem - configuration in, tested artifacts out - where fixes improve templates rather than generated code.

Simon Späti explores the architectural decisions required to build a declarative data stack engine (DDSE) that can generate entire data pipelines from configuration files. Drawing from his experience with data warehouse automation tools, he argues that separating business logic from technical implementation enables faster iteration, better testing, and maintainable data stacks. The article compares imperative versus template-driven approaches through code examples, examines whether existing orchestrators like Dagster could serve as the foundation, and surveys existing open-source tools in this space. He concludes that the path forward involves focusing on open standards and modular adapters rather than monolithic engines.

Key Insight: Semantic layers provide a structured foundation that makes them an ideal starting point for integrating GenAI capabilities into data infrastructure.

This is a teaser post introducing the third part of a series on semantic layers, focusing on their integration with AI for natural language data querying. The post argues that semantic layers may be the best starting point for GenAI initiatives in data organizations. The full article, published on Cube.dev, covers the architecture of AI-powered semantic layers and real-world use cases. The post positions semantic layers as a solution to the growing complexity and data governance challenges organizations face.

Key Insight: A universal semantic layer has become essential infrastructure for modern data teams, providing the single source of truth needed to maintain metric consistency while enabling broad accessibility across diverse tools and APIs.

This is a teaser post for a full article published on Cube.dev about universal semantic layers in modern data architectures. The post introduces the challenge data teams face with multiple sources, BI tools, and the need for consistent metrics. It references a prior article exploring semantic layers through an MVC lens and promises deeper coverage of how semantic layers evolved, their components, benefits like unified access and security, and integration capabilities with SQL, GraphQL, and Excel APIs.

Key Insight: Bluesky's open protocol and free APIs restore the developer-friendly, decentralized ethos of the early web, creating unprecedented opportunities for the data community to build collaborative tools on top of social network data.

Simon Späti explores Bluesky as an open-source social network built on the AT Protocol, highlighting how its open APIs and decentralized architecture return power to users and developers. He demonstrates practical examples of querying Bluesky data using DuckDB and Python, from reading posts via SQL to streaming the firehose into MotherDuck. The post emphasizes that Bluesky's approach mirrors the old web's decentralized ethos, where users own their data on personal servers rather than surrendering it to centralized platforms. He showcases the growing data community building tools around Bluesky and calls for collaborative development.

Key Insight: The MVC pattern's separation of concerns, proven over decades in web development, provides a useful mental model for understanding why semantic layers matter in modern data architecture.

This post introduces an analogy between the Model-View-Controller (MVC) pattern from web development and modern data architecture's semantic layer. Späti argues that just as MVC separates presentation from business logic through a model layer, data teams need a similar separation between data storage and presentation through semantic layers. The post serves as an introduction to a longer article published on Cube.dev, positioning semantic layers as the data world's equivalent to Active Record in Ruby on Rails. The core thesis is that proven software architecture patterns can solve similar challenges in data engineering around metric definition and maintenance.

Key Insight: DuckDB has evolved from a lightweight embedded database into a production-ready analytics engine that dramatically simplifies data architectures while delivering order-of-magnitude performance improvements across diverse enterprise use cases.

This comprehensive guide catalogs 15+ companies using DuckDB in production across five key categories: zero-copy virtualized SQL, lightweight single-node compute, pipeline processing, embedded interactive apps, and enterprise security. The post demonstrates DuckDB's versatility through concrete case studies showing dramatic performance improvements, from Watershed's 10x faster analytics to FinQore's pipeline reduction from 8 hours to 8 minutes, and Okta's processing of 7.5 trillion security records. The author argues that DuckDB's simplicity, embedded nature, and performance make it uniquely positioned to simplify data architectures across diverse use cases, from browser-based analytics via WebAssembly to replacing expensive cloud data warehouses.

Key Insight: Declarative BI-as-Code is the prerequisite for GenBI because AI models need explicit, machine-readable definitions of business logic and metrics - not opaque GUI configurations - to generate meaningful dashboards and analytics.

Simon Späti argues that Generative Business Intelligence (GenBI) represents a paradigm shift in how business users interact with data, enabled by the declarative BI-as-Code approach where dashboards and metrics are defined in YAML rather than hidden in GUI configurations. The key insight is that declarative definitions provide the semantic foundation AI models need to understand business logic, relationships, and metrics. GenBI promises to overcome the traditional bottleneck of data teams by allowing natural language conversations to generate dashboards and metrics. Späti positions this as the next evolution of self-service BI, with tools like Rill Developer already implementing these capabilities through LLM-powered dashboard generation.

Key Insight: DuckDB's unique position as an embedded analytical database enables enterprises to simplify infrastructure, reduce costs, and achieve faster insights by bringing compute closer to data and users.

DuckDB is an embedded analytical database that combines SQLite's simplicity with columnar-vectorized query execution for analytics workloads. The post categorizes DuckDB's enterprise value into five key use cases: interactive data apps, on-demand pipeline compute, lightweight SQL analytics, zero-copy SQL connector, and secure data handling. For enterprises, DuckDB offers significant benefits including reduced infrastructure costs, simpler deployment, and the ability to run complex analytics locally without network latency. The emerging 1.5-tier architecture, enabled by MotherDuck, allows the same DuckDB engine to run in both browser and cloud, fundamentally changing how analytical applications are built.

Key Insight: A truly declarative data stack would allow the entire pipeline to be recreated from configuration files alone, treating infrastructure like source code while cleanly separating the unchanging logic from the ever-changing data.

Simon Späti argues that the data stack is evolving toward declarative, code-first approaches where entire pipelines—from ingestion to visualization—can be defined in configuration files rather than imperative code. He traces declarative computing's history from Fortran to SQL to Kubernetes, showing how abstracting the 'how' empowers non-programmers while maintaining software engineering best practices. The post introduces a conceptual framework where a complete data stack could be recreated from a single function like `run_stack(serve(transform(ingest)))`, separating stateless configurations from stateful data assets. Späti acknowledges this unified declarative stack doesn't fully exist today but sees tools like Dagster, SQLMesh, and Rill as moving in this direction.

Key Insight: A well-designed personal knowledge management system should prioritize finding information through intuitive connections and associations rather than rigid folder hierarchies, using open formats that outlast any single tool.

Simon Späti details his comprehensive Obsidian-based note-taking workflow, explaining how he evolved from scattered notes across multiple apps to a unified 'Second Brain' system. He emphasizes using plain text Markdown files for future-proofing, combining PARA and Zettelkasten methodologies for organization, and leveraging Vim motions for efficient navigation. The workflow includes templates for different note types, a tag-based system for note maturity levels, and public sharing via Quartz. His core argument is that this system provides clarity, peace of mind, and the ability to find knowledge intuitively through graph-based connections rather than rigid folder structures.

Key Insight: Adopting Vim motions and Markdown as universal tools creates a compound productivity advantage that transfers across all text editing contexts, from code to notes to long-form writing.

Simon Späti recounts his journey from traditional editors like Notepad++ and VS Code to adopting Vim and Markdown as the foundation of his entire workflow. He describes how learning Vim's modal editing philosophy transformed his efficiency, comparing it to surgical precision with text. The transition from WordPress to static site generation with Markdown files eliminated formatting battles and gave him a single source of truth. Combined with Obsidian for note-taking (with Vim keybindings enabled), he now writes everything from notes to a full book in plain text, publishing with simple scripts.

Key Insight: Data modeling remains fundamentally about translating business semantics into data structures—a task requiring human domain expertise that AI cannot automate, despite advances in generative AI.

This final installment of Simon Späti's data modeling series examines how data architecture patterns influence modeling decisions, from the enduring staging-cleansing-core-mart layers (rebranded by Databricks as bronze-silver-gold) to specialized patterns like batch vs streaming, lakehouse vs warehouse, and semantic layers. The post surveys data modeling tools across commercial and open-source options, explores whether AI can automate data modeling (concluding it cannot replace human domain understanding), and introduces frameworks like ADAPT and BEAM for structured modeling approaches. Späti argues that despite tool proliferation and AI hype, the core challenge remains translating business semantics into data structures—work that requires deep domain knowledge and human judgment.

Key Insight: Dimensional modeling remains essential in modern data engineering because it forces teams to address fundamental questions about business value, granularity, and entities upfront—problems that will otherwise surface as mysterious failures in the analytics layer.

This second part of a data modeling series explores the approaches and techniques essential for structuring data effectively. The post contrasts top-down approaches (starting with conceptual models from business requirements) with bottom-up approaches (starting from existing physical data sources), recommending a combination of both. It surveys key modeling techniques including dimensional modeling (still highly relevant despite being from 1996), data vault modeling for handling big data and changing connections, anchor modeling for schema flexibility, and bitemporal modeling for historical data tracking. The author emphasizes that poor data modeling decisions manifest as problems in the 'last mile' of analytics, making early attention to modeling critical for project success.

Key Insight: Data modeling is experiencing a well-deserved resurgence because organizations have learned that skipping foundational data architecture in favor of quick MVPs leads to exponentially growing complexity and costs.

Simon Späti argues that data modeling is the overlooked foundation of data engineering that deserves renewed attention. He traces the history from Codd's relational model through Kimball and Inmon's data warehousing approaches to modern techniques. The post explains why data modeling fell out of favor—rushed MVPs, small teams under pressure, lack of time—and why it's resurging as organizations realize that solid data architecture determines the success of data-driven culture. Späti introduces key concepts including normalization, metrics/KPIs, slowly changing dimensions, and ERDs as foundational knowledge for data engineers.

Key Insight: Apache Arrow's emergence as the universal in-memory data format is reshaping the Python data ecosystem, making tool choice less consequential since zero-copy interoperability means data flows seamlessly between Pandas, Polars, DuckDB, and other Arrow-compatible systems.

Pandas 2.0 represents a major evolution in the Python data manipulation ecosystem by adopting Apache Arrow as its backend, replacing NumPy. This shift addresses long-standing issues with memory efficiency, speed, and interoperability that plagued earlier versions. Arrow's standardized in-memory format enables zero-copy data sharing between tools like Pandas, Polars, and DuckDB, dramatically reducing overhead in data pipelines. The article surveys the broader ecosystem of alternatives including Polars (Rust-based with query optimization), DuckDB (SQL interface), and Dask (parallelization), noting that Arrow's adoption as a common standard makes switching between tools nearly seamless.

Key Insight: The most meaningful life decisions come from training your intuition through experience and reflection, then having the courage to follow it even when it diverges from conventional expectations.

Simon Späti shares his personal journey of leaving the conventional 'Default Path' of career progression to forge his own 'Pathless Path' - a concept from Paul Millerd about embracing uncertainty over conformity. He traces how his supportive childhood, early passion for computers, sabbaticals in Australia and Bali, and relocating to Copenhagen shaped his identity. Along the way, he discovered stoicism as a guiding philosophy, fell in love with English as a creative medium, and ultimately found his life partner. The post concludes with his return to Switzerland, starting a family, and embracing remote work and writing - demonstrating that everyone's pathless path looks different.

Key Insight: The Modern Data Stack offers enterprises a path to escape vendor lock-in and gain competitive advantage, but adoption requires honest assessment of organizational capacity—integrating good tools doesn't automatically produce a good stack.

Simon Späti examines why mid-to-large enterprises struggle to adopt the Modern Data Stack (MDS), despite its open-source advantages. He traces the typical enterprise journey from custom scripts and legacy systems through the technology adoption lifecycle, noting that MDS tools are still in their 'early market' phase. The post contrasts US startup culture's rapid adoption with European enterprises' focus on cost, privacy, and regulatory compliance. While acknowledging the complexity and integration challenges of modular MDS tools, Späti argues that open-source solutions offer enterprises significant opportunities in flexibility, legal compliance, and competitive advantage—provided they have the technical capacity to manage them.

Key Insight: Vim's grammar-based approach to text editing - combining verbs, motions, and objects - transforms memorizing shortcuts into learning a composable language that makes you faster over time rather than hitting a productivity ceiling.

Simon Späti argues that Vim is not just an editor but a powerful language for text manipulation that every programmer should learn. He explains the grammar behind Vim - verbs, subjects, objects, and motions - which allows users to combine a few memorized commands into hundreds of combinations. After years of hitting productivity limits with standard editors, he discovered that Vim's modal editing approach optimizes for editing rather than writing from scratch. While acknowledging the steep learning curve, he shares how Vim transformed not just his coding but his entire workflow, from writing to browsing, making him a better developer through deeper terminal fluency.

Key Insight: The shift from custom data scripts to an open, composable data stack isn't just about using newer tools—it's about adopting battle-tested, interoperable components that enforce engineering best practices and eliminate the endless reinvention of foundational data infrastructure.

Simon Späti argues that the 'Open Data Stack' is a better framing than 'Modern Data Stack' because it emphasizes tools built on open source and open standards, which integrate better and reduce vendor lock-in. He presents four core tools as essential for any data-driven company: Airbyte for data integration, dbt for transformation, Metabase for analytics, and Dagster for orchestration. The post contrasts the old way of writing custom scripts for each data task with the modern approach of leveraging battle-tested open-source components. He emphasizes that this stack is modular and extensible, allowing teams to add specialized tools as needs grow.

Key Insight: Python-based configuration of data integration tools like Airbyte enables dynamic, event-driven pipelines that can adapt to changing metadata and external inputs, bringing infrastructure-as-code principles to data engineering.

This post demonstrates how to use Dagster's Python-based configuration to dynamically create Airbyte data integration pipelines, contrasting it with static YAML configurations via Octavia CLI. Simon builds a demo that scrapes GitHub repository links from an 'Awesome Data Engineering' list using Beautiful Soup, then uses Dagster to programmatically configure Airbyte sources, destinations, and connections to track stargazer trends. The key advantage is handling dynamic, event-driven scenarios where configurations depend on external factors that change over time. The post walks through the complete stack: Airbyte for data integration, dbt for transformations, Dagster for orchestration, and Metabase for visualization.

Key Insight: Rust won't replace Python for data engineers, but will increasingly power the core infrastructure underneath Python wrappers—much like what Scala was supposed to become.

Simon Späti examines whether Rust could become a significant language for data engineering, concluding that while Rust won't replace Python, it's increasingly powering the backend tooling that data engineers use. He highlights Rust's advantages—strong typing, compile-time error checking, memory safety, and performance—as particularly valuable for data engineering's many moving parts and unreliable data sources. The ecosystem is growing with projects like DataFusion, Polars, and Delta Lake Rust, but Python's massive ecosystem inertia means Rust will primarily serve as infrastructure beneath Python wrappers rather than as a primary language for most practitioners.

Key Insight: The semantic layer's resurgence stems from the explosion of data tools creating metric silos, and while it promises centralized, version-controlled business logic, organizations must weigh this against the operational complexity of adding another critical layer to their stack.

The semantic layer is a translation layer between data and business users that defines metrics once and syncs them across multiple tools (BI, notebooks, data apps). Unlike transformation layers that pre-calculate data, semantic layers operate at query time, offering flexibility without storing multiple table versions. The concept dates back to SAP BusinessObjects in 1991 but has resurged with modern tools like Cube.js, MetricFlow, and dbt Metrics. While it promises centralized business logic, reduced duplication, and better lineage, it adds operational complexity and maintenance costs. The semantic layer intersects with OLAP cubes, data virtualization, data mesh, and data catalogs, each serving related but distinct purposes.

Key Insight: Data lake table formats are the critical missing layer that transforms cheap, chaotic cloud storage into governed, queryable analytical platforms with database-like capabilities.

This comprehensive guide explains the differences between data lakes, data lakehouses, and data warehouses, focusing on how open-source table formats (Delta Lake, Apache Iceberg, Apache Hudi) transform simple cloud storage into powerful analytical platforms. The post details the three-layer architecture: storage layer (S3, Azure Blob, GCS), file formats (Parquet, Avro, ORC), and table formats that provide database-like features. Späti argues that data lake table formats are essential because they add ACID transactions, schema evolution, time travel, and unified batch/streaming capabilities to distributed files. The piece arrives at a particularly heated market moment, with Databricks open-sourcing Delta Lake 2.0 and Snowflake announcing Iceberg integration.

Key Insight: The future of data orchestration lies in treating data products as first-class declarative objects that business users can understand, rather than focusing on the technical details of task dependencies and pipeline execution.

Simon Späti argues that data orchestration is shifting from task-centric pipeline thinking to data product-centric approaches. Traditional orchestrators like Airflow focus on modeling task dependencies, but modern tools like Dagster and Prefect enable declarative, data-aware pipelines where data assets are first-class citizens defined ahead of runtime. This shift aligns with Data Mesh principles by making pipelines understandable to business users through a Data Product Graph that shows upstream dependencies and business logic. The post compares orchestration approaches, explains key abstractions (jobs, tasks, resources, triggers), and positions this evolution as essential for managing increasingly complex data environments.

Key Insight: A personal knowledge management system is not merely about organizing information—it's a framework for living deliberately, reducing cognitive load, and freeing mental space for what truly matters.

Simon Späti presents a comprehensive personal knowledge management (PKM) system that combines multiple methodologies including the Second Brain, Zettelkasten, and Getting Things Done. The post argues that a well-designed PKM system reduces stress and enables a 'deeper life' by externalizing thoughts and tasks into a trusted external system. Starting from minimalism and vim's distraction-free environment, Simon built a workflow incorporating smart note-taking, linked thoughts, and systematic task management. The deeper insight is that PKM is ultimately about life itself—what we value, how we learn, and who we want to become.

Key Insight: The next evolution in data engineering is building a unified Analytics API with GraphQL that centralizes metrics definitions, decouples API schemas from databases, and serves all stakeholders through one intelligent endpoint instead of fragmenting business logic across tools.

Simon argues that GraphQL can serve as the foundation for a unified 'Analytics API' that abstracts away heterogeneous data stores and provides a single interface for all data consumers. He outlines five key components: a GraphQL gateway/query engine, a metrics store (Headless BI), a data catalog, an orchestration layer, and a SQL connector. The post addresses how different stakeholders—ML engineers, BI users, power users, internal pipelines, and external customers—all need data access but through different interfaces. By centralizing metrics definitions and using GraphQL's built-in type validation, self-documentation, and schema decoupling, teams can avoid duplicating business logic across tools while maintaining flexibility to swap underlying data stores.

Key Insight: Own your notes by storing them as plain-text Markdown files locally - this gives you freedom from vendor lock-in and enables any future tooling or automation you might need.

Simon Späti explains his migration from OneNote to Obsidian after 10+ years of note-taking. His core requirement was data ownership through plain-text Markdown files stored locally, rejecting proprietary formats that lock users in. After evaluating Notion, Roam, Joplin, and plain Markdown, he chose Obsidian because it's free, works as a wrapper around local folders and Markdown files, and offers features like Graph View. He provides Python scripts to help others export their OneNote notebooks to Markdown, demonstrating the technical process of escaping proprietary note-taking ecosystems.

Key Insight: Treating blog content as code—version-controlled markdown files with local preview—removes enough friction that it actually makes writing fun again.

Simon Späti documents his migration from WordPress to Hugo, a static site generator. After years of struggling with WordPress plugin management, template incompatibilities, and the friction of converting markdown content to WordPress format, he found Hugo offered a simpler, git-based workflow. The migration was straightforward using an export plugin, and while page speed improvements were modest, the real value was in treating content as code—editable locally, version-controlled, and deployable with simple scripts. He describes the move as a 'homecoming' to the simplicity of his early web development days.

Key Insight: A modern data engineering project can be built entirely with open-source tools—from scraping to visualization—running cloud-agnostic on Kubernetes, with Delta Lake providing the database features needed on distributed files and Dagster orchestrating the heterogeneous components.

Simon Späti presents a comprehensive hands-on data engineering project using a real estate web scraping use case to demonstrate a modern open-source data stack. The project combines web scraping with Beautiful Soup, S3 storage via MinIO, Delta Lake for ACID transactions and schema evolution, Apache Druid for OLAP, Superset for visualization, and Dagster for orchestration—all running on Kubernetes. The post serves as both a practical tutorial and an architectural guide, showing how these tools fit together in the Databricks Lakehouse paradigm. He emphasizes cloud-agnostic design and provides all code in public GitHub repositories for others to learn from.

Key Insight: Modern data engineering techniques—data lakes, transactional processing, notebooks, and declarative pipelines—can transform opaque, slow BI systems into transparent, accessible platforms that empower all data-savvy users, not just BI specialists.

Simon Späti argues that traditional BI suffers from speed, transparency, and accessibility problems that data engineering approaches can solve. He presents twelve techniques including data lakes, Delta Lake for transactional processing, notebooks for democratizing access, Python/SQL pipelines, open-source tooling, incremental loading, and declarative programming. The post advocates for moving away from rigid data warehouse paradigms toward more flexible, transparent architectures that empower all data-savvy users. For organizations lacking resources, managed platforms like Ascend.io or Palantir Foundry offer these capabilities out-of-the-box.

Key Insight: Email's fundamental flaw is that it puts control of your attention in everyone else's hands, making deep work nearly impossible without deliberate structural and cultural changes to workplace communication.

Simon Späti argues that email is an outdated and inefficient communication method for modern workplaces, causing constant interruptions that destroy productivity and deep work. Studies show workers switch tasks every three minutes, largely due to email, with each interruption requiring 20 minutes to regain focus. The post advocates for replacing email with structured project management tools like Basecamp, Slack, or GitLab that centralize communication, reduce noise, and align with how millennials prefer to work. Späti emphasizes that changing communication isn't just about tools—it requires a cultural shift toward valuing focused, productive work over constant availability.

Key Insight: Modern data architectures benefit from separating concerns across specialized open-source tools—columnar storage for speed, code-based orchestration for maintainability, and lightweight visualization for accessibility—rather than relying on monolithic proprietary ETL platforms.

Simon Späti explores building a complete open-source data warehouse architecture using Apache Druid for storage, Apache Superset for visualization, and Apache Airflow for orchestration. He explains why Druid's column-oriented, distributed design excels at OLAP workloads despite SQL join limitations. The post advocates for moving transformation logic into code-based orchestrators like Airflow rather than traditional ETL tools. He concludes that while these open-source tools work well together, enterprise users may still want to invest in commercial visualization tools since that's what end users actually see.

Key Insight: Traditional OLAP cubes are hitting their limits in the cloud and big data era, and organizations must choose from four distinct replacement strategies based on their specific requirements for query speed, data volume, and architectural complexity.

Simon Späti examines why traditional OLAP cubes, particularly Microsoft Analysis Services, are becoming obsolete and explores modern alternatives. He identifies key limitations including lack of Azure cloud support, data size constraints, and the complexity of MDX query language. The post categorizes replacements into four groups: pure OLAP technologies (Druid, ClickHouse), cloud data warehouses (Snowflake, BigQuery), data virtualization tools (Dremio), and serviced cloud analytics (Looker, Sisense). He emphasizes that the choice depends on specific requirements around response time, data volume, and architecture goals, while also highlighting the critical importance of orchestrators and cluster-computing frameworks to complete the modern data architecture.

Key Insight: The location-independent lifestyle isn't about constant travel but about intentionally designing work around deep focus sessions while remaining open to growth through new environments and cultures.

Simon Späti shares his experience transitioning from traditional corporate work to a location-independent lifestyle, documenting his journey from Copenhagen to Bali, Southeast Asia, and across Europe. He argues that working remotely while traveling enables deep focus and natural productivity breaks, creating an ideal work-life integration. The post outlines the mindset and skills needed for this lifestyle, emphasizing creativity, continuous learning, and starting small. He reflects on lessons learned including the importance of cross-cultural connections, stoic philosophy, and maintaining a home base while embracing mobility.

Key Insight: Achieving deep work requires intentionally engineering your environment with the right tools—from scientifically-designed focus music to noise-cancelling headphones to email management systems that minimize distraction.

Simon Späti shares his favorite productivity tools across several categories: instant translation (Google Translate's camera feature, Skype Translator), grammar correction (Grammarly), email management (Google Inbox with its bundling and snooze features), focus music (Focus@Will, Brain.fm, Noisli), noise-cancelling headphones (Bose QuietComfort), quick notes (Google Keep), and Chrome extensions (Momentum, OneTab). The post emphasizes how these tools collectively help manage communication, reduce distractions, and enable deep work. He particularly highlights the science behind focus music, citing studies showing how specially designed audio can enhance concentration without the distraction of regular music.

Key Insight: Data engineering represents the natural evolution of BI engineering as organizations shift from batch-oriented, tool-driven ETL to code-first, real-time data pipelines that require software engineering skills alongside traditional data warehouse knowledge.

Simon Späti argues that data engineering is evolving from traditional data warehousing, driven by the growing need for real-time data, scalable pipelines, and programmatic approaches over drag-and-drop ETL tools. He traces his own journey from BI engineer to data engineering, explaining how the role combines software engineering skills with BI expertise and big data capabilities. The post details how ETL processes, data modeling practices, and even organizational education around data are all transforming. Späti concludes that while data warehouses will always be needed for structured business data, the methods for building them are shifting toward engineering-centric approaches, making data engineers the natural evolution of BI engineers.

Key Insight: Data warehouses and data lakes complement rather than compete—organizations need both, with data lakes for exploration and data warehouses for making insights accessible and structured for everyday business users.

This 2017 post clarifies the distinctions between data warehouses and data lakes, arguing against the prevailing notion that data lakes will replace data warehouses. Späti explains that data warehouses serve structured, slowly-changing analytical needs with ETL processes, while data lakes handle unstructured, fast-changing big data with ELT approaches. He advocates for maintaining both environments in modern organizations, using data lakes for exploration and data warehouses for structured, accessible reporting. The post also introduces Data Vault methodology as a bridge between the two paradigms and promotes Data Warehouse Automation tools as the future of DWH development.

Key Insight: The best note-taking system is one that lets you write first and restructure later, growing organically as your projects evolve.

Simon advocates for Microsoft OneNote as his go-to tool for organizing all aspects of life, from work projects to travel planning. He argues that OneNote's hierarchical structure of notebooks, sections, and pages allows notes to grow organically without losing organization. The post uses a two-month Asia trip as a practical example of how OneNote handles complex, evolving projects. He emphasizes that the tool's real power comes from long-term use, becoming a single repository for all essential information across work, personal life, and education.

Key Insight: Data-driven warehouse automation aligns well with agile methodology because it lets teams work with actual data and show real results to stakeholders quickly.

This 2017 post surveys the data warehouse automation (DWA) tool landscape, comparing market presence using Google Trends data. Späti distinguishes between model-driven approaches (design first, then find data) and data-driven approaches (generate models from available data). He references external resources including Hans Michiels' comprehensive comparison guide and survey results. The post notes that most vendors lack transparent pricing, with BimlFlex being a notable exception. WhereScape and TimeXtender lead in search volume, though the author acknowledges search data conflates company and product names.

Key Insight: Data warehouse development is fundamentally repetitive work that follows predictable patterns, making it an ideal candidate for automation that frees teams to focus on actual data analysis rather than technical plumbing.

Simon Späti makes a case for Data Warehouse Automation (DWA) as the next logical step in an industry where automation has transformed every sector. He argues that in an era where data is the new gold and time is precious, manually building data warehouses creates unnecessary friction and errors. DWA enables teams to deliver KPIs within weeks rather than months, shifting focus from technical plumbing to actual data analysis. The core value proposition is eliminating repetitive tasks like creating staging tables, managing schema changes, and handling slowly changing dimensions—work that follows predictable patterns and begs for automation.

Key Insight: Data warehouse automation is essential because traditional manual development can no longer keep pace with modern business demands, and DWA tools will restore the relevance of data warehousing in an era dominated by big data hype.

Simon introduces a blog series on Data Warehouse Automation (DWA), arguing that while the fundamental DWH architecture hasn't changed much since IBM created it in 1986, the approach to building data warehouses must evolve. He contends that traditional manual approaches are too slow for modern business needs, and DWA tools represent the future of the discipline. The post serves as a table of contents linking to deeper explorations of why DWA isn't more popular, its benefits, and available tools.

Key Insight: Data Warehouse Automation isn't about replacing developers but freeing them from repetitive implementation tasks to focus on the genuinely challenging work of data modeling and business requirements.

Simon Späti advocates for Data Warehouse Automation (DWA) tools, arguing they dramatically accelerate delivery while maintaining consistency and quality. He addresses common objections including fear of losing control, vendor lock-in, need for skilled personnel, licensing costs, and speed concerns. Drawing from his experience on a year-long project delivering 162 tables in three months, he demonstrates that DWA enables parallel development and faster iterations impossible with traditional methods. He concludes that as businesses demand faster insights and agile delivery, DWA adoption is inevitable despite emotional resistance from developers fearing replacement.

Key Insight: Small utility tools that run quietly in the background—tracking time, managing clipboards, storing passwords—compound into significant productivity gains when used consistently.

Simon shares his personal toolkit for productivity and daily computer use, covering six categories of tools: time tracking (RescueTime), clipboard management (Ditto/Paste), password management (LastPass/KeePass), screen capturing (Snagit), blue light filtering (f.lux), and bookmark syncing (Xmarks). He emphasizes that these tools have genuinely improved his workflow and encourages readers to try them. The post reflects his practical, hands-on approach to optimizing work habits through technology. He notes this is part one of a series, with more tools to follow.

Key Insight: Freely sharing knowledge creates a virtuous cycle where generosity is reciprocated with more input, stronger relationships, and collaborative innovation.

Simon reflects on a personal experience from his apprenticeship where he learned the value of sharing knowledge freely. Initially reluctant to share exam summaries he and a friend created, he eventually opened up and discovered that sharing led to receiving more input, better relationships, and valuable discussions. He connects this lesson to the rise of open source software and encourages readers to embrace a sharing mindset.

Key Insight: Database migrations between Oracle and SQL Server require careful attention to function behavior differences, especially around numeric precision and rounding, where seemingly equivalent functions can produce different results.

This technical guide documents the challenges of migrating Oracle database views to Microsoft SQL Server, focusing on function compatibility differences between the two platforms. The author provides a comprehensive reference table mapping Oracle functions to their SQL Server equivalents, highlighting cases where direct translations don't exist. Special attention is given to numeric conversion pitfalls, particularly how TO_NUMBER, CONVERT, and CAST handle decimal rounding differently. The post includes practical SQL examples demonstrating these differences and provides custom SQL Server functions that replicate Oracle behavior for TO_CHAR, TO_NUMBER, MOD, INSTR, and SUBSTR.

Key Insight: When migrating Oracle recursive queries to SQL Server, inline user-defined functions with temp tables can overcome the performance limitations of CTEs for complex nested views.

This post continues a series on migrating from Oracle to Microsoft SQL Server, focusing specifically on performance challenges with recursive queries. The author encountered significant issues translating Oracle's optimized SYS_CONNECT_BY_PATH function to SQL Server CTEs, especially in complex nested views. The solution involved using inline user-defined functions and temp tables to dramatically improve query response times from non-responsive to 1-5 seconds. The approach maintained transparency for end users and applications by wrapping functions in views.

Key Insight: Successful database view migrations require systematic validation of both performance metrics and data content accuracy using automated comparison scripts.

This technical post concludes a three-part series on migrating views from Oracle to SQL Server, focusing specifically on testing methodologies. The author demonstrates how to verify both performance and data correctness after migration using SQL scripts. For performance testing, he recommends SELECT COUNT(*) queries with cache clearing, providing a script to auto-generate these statements. For content comparison, he explains using linked servers and EXCEPT queries to identify discrepancies between Oracle and SQL Server view outputs.

Key Insight: Purpose-designed background music may be more effective for concentration than mainstream music because it avoids the emotional and intellectual engagement that causes distraction.

This brief post recommends Focus@Will, a music streaming service designed to enhance concentration while working or studying. The author shares that the service uses neuroscience-based music channels that aim to engage the brain's limbic system without causing distraction. The post includes marketing claims about the science behind the approach, citing partnerships with neuroscientists. Simon personally endorses it based on his experience using it during his bachelor's degree work.

Key Insight: Quality noise-canceling headphones are a worthwhile investment for knowledge workers dealing with distracting environments.

A brief product recommendation for Bose QuietComfort noise-canceling headphones. The author endorses these headphones for anyone who travels frequently or works in shared office spaces with multiple people. While acknowledging the high price, he argues the active noise cancellation technology makes them worth the investment for blocking out ambient sounds like trains, flights, and conversations.

Key Insight: A linked server bridging T-SQL and XMLA enables automated SSAS cube partition management that would otherwise require complex SSIS packages.

This technical post addresses the challenge of dynamically generating partitions for SSAS (SQL Server Analysis Services) Cubes, which lacks built-in easy partition management. The author presents a SQL-based solution that uses a linked server to execute XMLA statements from T-SQL, enabling automated monthly partition creation. The approach involves creating stored procedures that generate and execute XMLA commands to delete and recreate partitions based on date ranges. This allows for programmatic cube partition management without relying on SSIS packages.

Key Insight: Adblockers are a simple solution that makes web advertising a non-issue you'll eventually forget exists.

A brief post recommending adblockers to eliminate distracting website advertising. The author notes they've been free from ads for 2-3 years thanks to browser extensions, and encourages readers to adopt the same approach. Two specific adblocker tools are recommended along with a comparison resource.

Key Insight: Database collation settings cause ORDER BY to behave differently between Oracle and SQL Server, a subtle gotcha when migrating between platforms.

This brief technical post highlights a subtle but important difference between Oracle and Microsoft SQL Server: the ORDER BY clause behaves differently due to database collation settings. The author demonstrates that identical text data sorted in Oracle versus SQL Server produces different results unless you explicitly specify a compatible collation. By using SQL_Latin1_General_CP850_BIN2 collation in SQL Server, the ordering can be made to match Oracle's behavior.