If you have any ideas for improving this investigation, I'm all ears!

----------------------------

Since reading the OpenAI data stack post, I've suspected that dbt/SQL might get in the way of LLMs when looking at the data stack more holistically.

By data stack, I'm talking Modern Data Stack: ETL core app data into Snowflake/BigQuery, load other API data like Stripe in as well, do SQL joins to get answers (if unfamiliar then this post might not be that clear).

All the SQL + metadata might just be more human useful than LLM useful.

Or the system might have been better designed for when we didn't have Coding Agents.

I've wondered about this a bit:

On a related note to dbt, a strongly coupled feeling, I've felt that your main app's code is underutilized as a source of meaning and structure:

Then I see this point from Open AI, and I'm like, YES!

I've felt this acutely. Maintaining SQL files in dbt has just so much surface area, and so little logic!

But also reading the OpenAI post, I see they are still running most of their analytics logic in SQL!?

I really struggled to believe that the OpenAI data, at the fastest-growing, most well-funded supercompany in recent memory, is doing exactly what I would do.

They are running the Covid era MDS at the core of all the other stuff.

Same way, same tools, driven by the same FinOps need.

I don't expect SQL to go anywhere, that is not what I'm getting at. I also don't think dbt should go anywhere necessarily. Standards are set in times of disruption, and if dbt is the analytics standard then so be it.

But I wanted to scratch the itch. So I built a benchmark.

I present The Modern Data Benchmark (or MDS Gym?).

A small experiment comparing how LLM agents perform across different data architectures, given the same data and the same questions.

The pro-forma result:

Across 7 LLM models, warehouse+dbt had a 5% pass rate. App-unified architectures had 38-48%.

*the numbers in this post are directionally correct, like the MDS.

A quick history of how we got here

Before we dive into it, some context.

The Modern Data Stack came out of a specific organizational need.

1. Marketing needs to track ad spend.

2. Finance needs to reconcile Stripe revenue.

These are important but non-core functions, so they get staffed by analyst-operator types who can usually write SQL but not Python. Add Redshift/Snowflake and all roads lead to you SQL.

dbt emerged to give those SQL queries just enough software engineering discipline, version control, modularity, templating, without forcing anyone to leave SQL. It was a rational solution to a real constraint: these teams could not write very good code, and didn't have a place to write it.

The gravity of that constraint pulled everything towards raw SQL against a data warehouse. No types, no abstractions, nothing but SQL. dbt came along to solve the obvious shortcomings (shitshow) of lots of SQL. Version Control+Jinja and it was, I was working in enterprise data before, dbt was wild!

OpenAI's data warehouse confirms the pattern

What surprised me was looking at OpenAI's data warehouse. On the surface it looked sophisticated: context layers, embeddings, all the works. But at the core, two things stood out.

First, it was driven by FinOps. The example given was revenue reconciliation from Stripe. Even at frontier AI companies, FinOps drives the data warehouse.

Second, it uses SQL (and I guess dbt). The team likely had used dbt before, so they reached for it again. This isn't a criticism, it's how standards form. Not by systematic evaluation, but by repetition.

The question worth asking

dbt's value was making SQL manageable for human analysts. But at scale there are now tens of thousands of lines of SQL that no human is ever going to read. The SQL is increasingly being consumed by agents.

If the consumer is an agent, "easiest for humans to read" stops being that important.

The relevant question becomes:

For an agent answering business questions, which representation of the system is the most legible, robust, and correct?

It all felt rather benchmarkable.

The split-brain problem

The modern data stack creates a structural split. Your app has your core business logic: users, statuses, transactions. Separately, you have a data warehouse that holds a lagging copy of that data, plus third-party data like Stripe that only exists in the warehouse.

To answer anything useful, you need to join app data onto Stripe data inside the warehouse, using SQL, with constrained logic. The "single source of truth" in the warehouse is never truly trustworthy. Your actual source of truth is the production database and someone else's API, and the warehouse is always behind.

The crux: what if you brought the data closer to home? Shift left? Strongly typed? Asked Codex 5.3 to spar with Opus 4.6 on turbo mode? What would they do?

I guess they'd load Stripe data into a structure your app understands, with proper types and constraints, and they'd run analytics against the unified codebase..?

If revenue is a key business capability, and we're no longer as code-constrained as we were, why not model it as a first-class concept?

(this has a million small holes, but stay with me)

The benchmark

Three sandbox environments, same data, same three analytical tasks: ARPU, churn rate, and LTV.

Small data, known correct answers. Size isn't the test. We are looking at architecture.

In the app sandboxes, Stripe data is represented as internal data with types.

App Tables + Stripe Tables → Single typed context → Code → Metric

In the dbt sandbox, it follows the traditional pattern: third-party data loaded and joined via SQL.

App DB + Stripe → Replication → DuckDB raw tables → Staging SQL → Marts SQL → Metric

The model must discover the schema and produce executable code that returns the correct number. No hints, no hand-holding.

The key difference: in app architectures, the model has one typed context. In the warehouse, it must navigate staging models, column naming conventions, and SQL casting to arrive at the same answer.

How the evaluation works

Each run works like this:

1. Fresh sandbox. A clean copy of the sandbox template is created with the synthetic data loaded. No prior work carries over between tasks.

2. Agent loop. The model gets a system prompt describing the architecture and four tools: read_file, write_file, list_files, and done. It has up to 10 turns (API round-trips) to explore the codebase, discover the schema, write its solution, and signal completion. Temperature is set to 0.

3. No hints. The model is told what to compute (e.g., "ARPU for active users") and given the function signature, but not how. It must figure out join keys (users.stripe_customer_id → invoices.customer_id), column names, and time anchoring on its own by reading files.

4. Execution app-typed: the TypeScript function is imported and called with the data arrays.

5. Execution app-drizzle: the async function runs against a pre-loaded SQLite database via Drizzle ORM.

6. Execution warehouse-dbt: the SQL is executed in DuckDB with raw tables created from JSON (staging/mart views are built from any SQL files the model wrote)

7. Scoring. Pass/fail is purely numeric: does the output match the expected value within tolerance? (±1 for integers like ARPU/LTV, ±0.001 for rates like churn). No partial credit, no style points. If the code crashes, it's a fail. If it returns the wrong number, it's a fail.

8. Flexible matching. The validator accepts naming variations (calculateARPU, computeArpu, getArpu, etc.) and searches multiple directories for SQL files, so models aren't penalized for reasonable naming choices.

Expected values are computed from the same data by a reference implementation in the benchmark harness itself, not hand-coded, so they're guaranteed consistent.

THE RESULTS

The ORM sandbox was the clear winner: Opus got a perfect 3/3 every single run, and cheaper models like Kimi and Grok matched it.

The warehouse-dbt column is almost entirely zeros. Only Opus managed a single pass, and even that was inconsistent across runs.

Looking closer at variance on multi-run stability (n=5 per Anthropic model)

• Opus: Zero variance.

• Sonnet: High variance (~0.9 std).

• Haiku: Fluctuates between 0-1 passes.

Mid-tier models are at the edge, and the architecture is what pushes them over or pulls them back.

Where dbt struggles

The failure modes:

• Schema mismatch: wrong column names (created_at vs usage_created_at, org_id vs organization_id). Staging conventions that the model has to guess.

• Type mismatch: interval math on VARCHAR timestamps without casting.

• File naming: incorrect output filenames or failure to write the metric model.

Even thorough models struggle with dbt

Adding in a measure of unique files read per task in warehouse-dbt:

• Haiku: 1.1 files (barely looks at the schema)

• Sonnet: 3.7 files (reads staging files but still gets column names wrong)

• Opus: 4.4 files (reads everything, still only 1/3 pass rate)

Even when the model does its homework, the warehouse architecture introduces enough indirection to trip it up. It's not a laziness problem, the representation has too many seams.

In the app sandboxes, failures were simpler (wrong join key, missing function) and more recoverable in typed code.

What I take from this

This benchmark tests a narrow but important thing: can an agent read a schema, write executable logic, and return the correct metric? It doesn't test full dbt workflows with Jinja, ref/source, or materializations. It tests the core analytical task that everything else is built to support.

A few observations (not conclusions, this is early and the sample is small):

1. The architecture matters more than the model. The same model that fails at dbt can succeed in a typed environment.

2. ORMs are surprisingly agent-friendly. Drizzle over SQLite was the strongest sandbox, even mid-tier models could navigate it. Typed schema + query builder + unified context seems to hit a sweet spot.

3. Indirection has a cost that compounds. Each layer of staging, naming convention, and type casting is a place where an agent can silently go wrong. Types and co-location seem to reduce that surface area.

What's next

The current tasks (ARPU, churn, LTV) are intentionally simple, canonical SaaS metrics on synthetic data. The architecture signal is clear, but the questions need to get harder to be convincing.

A few directions:

1. What is a "fair" dbt project? After the initial results I started adding hints to the dbt sandbox to get some passing runs, things like cast annotations in staging models so the model doesn't trip on DuckDB timestamp arithmetic. I added CAST(created_at AS TIMESTAMP) AS usage_created_at in the staging layer as it kept tripping up on that. It sort of helped: adding a single cast hint let Sonnet pass org_churn_rate where it previously crashed on a runtime error (details), but it wasn't that consistently helpful. It also felt like a slippery slope. How much documentation and scaffolding do you add before the dbt sandbox stops being representative of what an AI agent would setup. Remember this was all setup by Codex 5.2, I didn't touch a thing! A real dbt project lives somewhere on this spectrum, and where exactly is an open question. But maybe that's the point. You can keep adding scaffolding to SQL, cast hints, schema docs, Jinja templating, ref() pointers, semantic YAML, and each one closes a small piece of the gap. At some point you have to ask: is the SQL architecture, with all the scaffolding you need to make it work for agents, converging on the thing you'd build if you just started with types and a unified codebase?

2. Realistic drift. The current benchmark is static, the data is clean and complete. Real analytics is messier: late-arriving Stripe invoices, missing stripe_customer_id mappings, schema changes mid-pipeline. Adding sync delay scenarios would test whether the split-brain problem is as bad in practice as it is in theory.

3. Linting as agent feedback. Early experiments with TypeScript typecheck + SQLFluff showed that giving agents lint feedback and extra fix attempts improved ORM more than dbt scores, but the improvement might just be from extra turns, not the lint signal. SQLFluff style rules seem to add noise that distracts smaller models. A schema-only mode that only surfaces missing tables/columns could be a cleaner signal. I tried some things there, but nothing clear.

4. Information parity. A typed codebase inherently carries more structural information (types, constraints, relationships) than raw SQL with YAML docs. You could argue that's confounding. I guess so? But that's also the point: the architecture is the information density. Still, enriching the dbt sandbox with comprehensive YAML schema docs would test how much of the gap is "types help" vs "unified context helps."

5. Turn-level analysis. Currently only tracking file-read counts. Understanding the step-by-step reasoning, where models go wrong, when they recover, would give sharper insight into why architecture matters.

Other interesting things:

• Semantic layer sandbox (Drizzle-Cube). The baseline benchmark included Drizzle-Cube but the architecture benchmark didn't. Adding it would test whether pre-defined measures help or constrain agents. I'm hopeful this pushes Drizzle well beyond comparison!

• Does the "context layer" exist? There's a lot of hand-waving right now about "context layers" and "context graphs." When you boil these down, they often look like a data warehouse or semantic layer in new language. My position: if you can't demonstrate a simple instance of a complex idea, it doesn't meaningfully exist. Next step is to build sandboxes for the best-case context graph blogs and run them through the same benchmark.

• Harder queries. @matsonj's platinum set from the BIRD text-to-SQL benchmark covering complex joins, CTEs, NULL handling, and conditional aggregation. Also KramaBench which tests full data pipelines, not just single queries, and from the benchmarks referenced by Sphinx including DABStep (real Adyen payments data). If agents struggle with simple ARPU, what happens with real analytical complexity?

• Costs. I tracked the costs, the results were somewhat interesting, it seemed like Opus was actually often cheaper as it took fewer laps to get the answer. I need to benchmark this more carefully. Pareto performance cost curve.

A request

I'm no longer that close to dbt. Things may have moved on. If you're actively working in dbt and you look at the warehouse sandbox and think "that's not how we'd set it up," I genuinely want to hear that. Is this a realistic task? Is this a fair test? The benchmark is open source, you can set up the dbt sandbox the way you think it should be, and run the same evaluation. If a well-configured dbt project closes the gap, that's a finding worth publishing too.

Caveats:

1. Small n, synthetic data, single-pass runs for some models, no full dbt compilation. This is directional, not definitive. Run it yourself, add harder tasks, prove me wrong.

2. Claude wrote this out from a voice-note I recorded.

Link to benchmark repo · Link to OpenAI data stack post

Over the Christmas break, I decided to explore code execution for AI agents, inspired by Anthropic’s blog on the topic. The idea is appealing: reduce the amount of unnecessary context that gets fed into an agent’s working memory.

As MCP usage scales, there are two common patterns that can increase agent cost and latency: Tool definitions overload the context window; Intermediate tool results consume additional tokens.

If you’ve spent any time with Claude Code or similar tools, you’ll know the problem. You really want to avoid the orchestrator seeing unnecessary logs, digging through dense files with low information density, or accumulating cruft that poisons the context window.

Anthropic’s approach uses an orchestrator that composes agent tasks without ever seeing the results of those tasks directly. It only sees structured outputs if the tool deems it necessary. I wanted to understand how this actually works, particularly for small local models. Can you get a 7B model to operate coherently well beyond its context limits, and can the orchestrator agent be useful without needing to see all the tokens?

The short answer is yes, sort of. But the interesting part is what that requires: tool design that makes composition obvious through what I’m calling tool ergonomics (python types) alone.

Starting Simple: Bash Scripts and Local Models

I started basic, using a local Llama 7B model. My first attempt was a simple feedback loop:

Writer (draft) → Editor (feedback) → Writer (revision) → ...

The editor provides critique on structure, pacing, and prose quality. The writer interprets the feedback and revises. In theory, this preserves the writer’s voice across iterations.

Neither this nor the direct rewrite variant worked particularly well. The feedback loop used about 3x the tokens per round, while direct rewrite was around 1.5x. But it wasn’t much better than just loading one model and hitting the 8k context limit. I wanted to stay around 2k tokens per model run to keep things focused (they seem happier there?).

The Iteration Loop (and Why It Failed)

Next, I built an iteration loop with a planner, writer, and critic. The problem became obvious quickly. The orchestrator was still seeing all the context:

Iteration 1: prompt → draft (2000 tokens)
Iteration 2: prompt + draft + critique → revised (4500 tokens)
Iteration 3: prompt + draft + critique + revised + critique2 → ... (growing)

Four or five iterations and it fell apart. No meaningful reduction in context bloat.

Trying smolagents (and Moving On)

I gave Hugging Face’s smolagents a go. The hope was that the LLM generates code, data flows through variables, and you get less context bloat.

The issue: smolagents uses ReAct (step-by-step reasoning), so the orchestrator maintains a memory of previous actions and observations at each step. My impression was that this meant tool outputs were still accumulating in context, making it worse than my bash scripts due to orchestrator overhead.

(Disclaimer: I didn’t rigorously measure this. I moved on fairly quickly because I wanted to build something from scratch that I understood fully. smolagents may well have optimisations or configuration options I missed. Take this with a grain of salt.)

Building My Own Orchestrator

This is where things got interesting. I (Claude) built an orchestrator that looks at a manifest of tools and writes code to invoke them. The key difference: the orchestrator generates the code once, then it runs in Python without the orchestrator seeing intermediate results.

def solve_task(user_prompt):
    draft = writer(prompt=user_prompt)
    for i in range(4):
        critique = critic(content=draft, requirements=user_prompt)
        if "DONE" in critique:
            break
        draft = writer(prompt=user_prompt, feedback=critique)
    return draft

result = solve_task(user_prompt)

The orchestrator only ever sees the first call and the final return. Everything in between is blind to it. This is what makes extended generation possible: the orchestrator’s context stays constant regardless of how many iterations run.

This also opens up interesting possibilities around data privacy. Imagine processing bank statements where you don’t want the full statement in context. A tool returns just the insight: “the largest customer is X”. That gets passed to the next tool, which finds that customer in the accounting system, but it doesn’t return their details, just the necessary step for analysis (their payment terms), and those aren’t shown to the orchestrator, they go to the underwriting tool, and so on. The orchestrator never sees the raw data, just structured results flowing between steps.

This isolation pattern has side effects: It’s the exact mechanism that was used to run Claude Code as a hacker. As Anthropic reported, attackers “broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose.” The orchestrator’s blindness enables privacy and extended context, but it also means the model can’t reason about the broader implications of what it’s being asked to do. Something to keep in mind when designing these systems.

The Complexity Valley

But then I hit what I started calling the “complexity valley”. As I added more tools (writer, critic, planner, outliner, evaluator…), the generated code became a mess of manual state tracking:

def solve(prompt, emphasis=""):
    chapters = planner(prompt)
    combined = ""
    outline = ""

    for i, ch in enumerate(chapters):
        prev = combined[-1000:] if combined else ""
        remaining = chapters[i+1:]
        remaining_str = "\n".join([f"- {c['title']}: {c['summary']}" for c in remaining])
        # ... and on and on

Small models couldn’t generate this reliably. They’d forget to update the outline, mishandle the slicing, or botch the string formatting.

The Template Trap

My first fix was to make the code generation prompt extremely explicit:

PROMPT: Explore - write {explore_chapters + 1} chapters organically.
Pattern:
  content = write(prompt, emphasis)
  contents = [content]
  for _ in range({explore_chapters}):
      next_prompt = explore(content, prompt)
      content = write(next_prompt, emphasis)
      contents.append(content)
  return combine(contents)"""

This worked, but is it cheating? If we have to show the exact code pattern, the model isn’t composing tools; it’s copying templates. Maybe?

The Discovery: Types Guide Composition

After multiple iterations, it started working best when tools have type-aligned signatures. The composition becomes obvious without explicit templates.

write(prompt: str) -> str           # string in, string out
plan(prompt: str) -> list[str]      # string in, list out
explore(content: str, original_prompt: str) -> str
combine(contents: list[str]) -> str # list in, string out
evaluate(content: str, original_prompt: str) -> str

When plan() returns list[str], the model knows it needs to iterate. When combine() takes list[str], the model knows to collect results. The generated code became correct without explicit patterns:

def solve(prompt, emphasis=""):
    chapters = plan(prompt)
    contents = [write(ch) for ch in chapters]
    return combine(contents)

A 1.5B parameter model could figure this out based on the types.

Naming mattered too. I changed explore(content, prompt) to explore(content: str, original_prompt: str). The name “original_prompt” signals “grounding context”. Models stopped inventing new prompts and started passing the original variable.

Early versions used rich dataclasses, but small models struggled with attribute access. The fix was just using strings. This allowed a 7B model to use it correctly 90-95% of the time, and a 1.5B model almost as well.

When It Goes Wrong

The most common issue: the model would overwrite content rather than appending.

# What the model should generate:
contents = []
for ch in chapters:
    content = write(ch)
    contents.append(content)

# What it actually generated:
for ch in chapters:
    contents = write(ch)  # Overwrites each time!

This happened less once the types were clear (list[str] signals “collect these”), but it never went away completely. Small models make small mistakes, and without a verification loop, garbage propagates.

[PLACEHOLDER: Include before/after example of generated text quality, the challenge is these are long passages]

Results

By the end, an 8B writing model paired with a 7B instructor model could work pretty seamlessly. Sometimes down to 1.5B for simple tasks.

I’d say it went from a 2/10 initially to maybe a 3-5/10 by the end. It would write a chapter, review it, write the next with tolerable handover, and string together coherent sequences. Not great literature, but the mechanics worked: small models running well beyond their context limits because the orchestrator carried zero burden.

I experimented with two modes: “explore” (write freeform, let each chapter lead to the next) and “plan” (outline upfront, fill in the gaps). Explore worked better. The modal split feels like a hack, but maybe that’s fine. Different mental models want different tool behaviour.

What I Learned

The thing that surprised me most was how much time went into tool design. Figuring out the most ergonomic way for tools to be usable by an agent, making them implicit and obvious and composed and intuitive. That was the work.

The best tools I created:

• Use strings, not custom objects
• Signal iteration with types (list[str] means “iterate over this”)
• Keep content out of orchestrator context
• Name parameters semantically (original_prompt not prompt)
• Validate ruthlessly, because small models make small mistakes

The insight: tool design for LLMs is about ergonomics. If the types make composition self-evident, if the tool is easy to use right and hard to use wrong, small models can orchestrate complex workflows.

Next Steps

Better continuity management. I’m figuring out how to manage continuity more explicitly, which led to tools that work differently in explore vs plan mode. This feels like a hack, but maybe matching tool behaviour to user mental model is actually correct.

Better verification. Right now, if the generated code is wrong, the output is garbage. I need lightweight checks before execution.

Testing the Bitter Lesson. As models improve, does the scaffolding help or hurt? Worth checking whether this structure earns its keep on larger models or just adds overhead.

One caveat to close on: this whole approach might age badly. As Peak noted, agent harnesses can limit performance as models advance. The structure improves performance, but this structure can limit performance as compute grows.

For small local models, making the orchestrator blind to intermediate results while letting types guide composition seems to work.

Appendix

The Final Architecture

┌─────────────────────────────────────────────────────────────┐
│  manifest.json (Public API)                                 │
│  - 5 tools with type signatures                             │
│  - Descriptions guide model selection                       │
└─────────────────────────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│  orchestrator.py (Code Generation)                          │
│  - Reads manifest                                           │
│  - Generates Python solve() function                        │
│  - Validates: forbidden patterns, param names, mock exec    │
└─────────────────────────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│  tools.py (Implementation)                                  │
│  - High-level: write, plan, explore, combine, evaluate      │
│  - Low-level: writer, critic, planner (internal only)       │
│  - TOOL_NAMESPACE exports only public tools                 │
└─────────────────────────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│  Generated Code Execution                                   │
│  - Content flows between tools                              │
│  - Orchestrator never sees intermediate results             │
│  - Constant context regardless of iterations                │
└─────────────────────────────────────────────────────────────┘

The manifest:

{
  "tools": [
    {
      "name": "write",
      "signature": "write(prompt: str, emphasis: str = '') -> str",
      "description": "Write content with automatic revision."
    },
    {
      "name": "plan",
      "signature": "plan(prompt: str) -> list[str]",
      "description": "Break a prompt into chapter summaries."
    },
    {
      "name": "explore",
      "signature": "explore(content: str, original_prompt: str) -> str",
      "description": "Generate next chapter prompt from current content."
    },
    {
      "name": "combine",
      "signature": "combine(contents: list[str]) -> str",
      "description": "Join chapters into final output."
    },
    {
      "name": "evaluate",
      "signature": "evaluate(content: str, original_prompt: str) -> str",
      "description": "Evaluate content against requirements."
    }
  ]
}

The analysis tracks three frameworks for mapping the cyclical nature of cultural phenomenon (one two three). These don’t map perfectly, and number three doesn’t fit the enterprise context. The idea: Something new catches on, it grows big, it loses its way, there is a fight / collapse / phase-change and then it stabilises. The subheadings are from John Cutler:

Anything helpful will eventually become commodified, industrialized, and watered down to the point of being unrecognizable. It happened with Agile, and is happening with strands of product management, DevOps, design, etc.

Phase 1: Precycle

MDS EMERGES FROM PRACTICE

People like solving problems that resonate with other people, open source vibes, this was the early MDS scene.

Before MDS, there were enterprise data vendors and mostly they weren’t very useful for fast moving companies, in the sense that you could only afford them if you budgeted in the $xx millions and you planned in half decades. Redshift precipitated a change and Snowflake accelerated it. Pay-per-query.

Redshift and Snowflake laid the soil that led to the Modern Data Stack. They created the easiest to buy large scale data storage that anyone had ever seen. The problem now became keeping track of the transformations necessary to deal with the vast data recycling centres that were suddenly viable to be created by smaller and smaller teams.

dbt, a relatively generic idea (SQL templating in python) became the foundation for a new movement to deal with the issue, right place, people, audience, right time.

I was instantly hooked. dbt took the lead among an array of relatively fragmented solutions all jockeying around the task at hand.

MDS NAME COINED / FIRST BLOGS

A brief google and it seems like dbt first used the term "Modern Data Stack” on January 29, 2018, (I couldn’t be bothered to dig any deeper, history in the form of a sea-shanty with a broken banjo).

The concept caught on. I was working as a Data Engineer doing Redshift transformations, dbt achieved this in a far better way. Before that, doing this stuff was super expensive, testing and version control was tricky, everything was slow.

Blogs came out describing the way. Here I generously quote myself describing the series of blog posts that introduced the new way

We, the desperate, listened closely. The message I heard: bring the best of software development to startup data analytics

I was likely referring to this blog post. It all sounds rather exciting in retrospect, and it was!

MDS MINDSET / MANIFESTO / PRINCIPLES / EVANGELISTS

The whole thing coalesced (yup) around the Community. dbt slack and Locally Optimistic slack were the absolute centre(s) of the data world for a fair amount of time.

When those inside the cutting-edge scenes band together to support, teach, and create with each other, their niche and experimental projects can become the new normal on top of which the next generation builds.

The vendor guidelines kept things civil in the dbt chat, and Locally Optimistic was generally smaller and less frantic. Dbt was often the target of staffing firms who would point their junior devs at the dbt slack and say “here is your technical support” and they would paste 200 line error logs into the main chat and say “what do??” and then disappear.

I think a real kicker was the Open Source hook. Everyone wants to work with open source software. Now data people could do that!

There were reams of Guides, here is mine. (filler content for Dataform, now acquired by Google). The writing wasn’t very good in hindsight, but I stand by my suggestions, move fast with simple tools. Here is another good guide. These ideas were evaluated on some or other believability and the good ideas were amplified.

Alongside this, Validation that the data trade was noble fanned the flames.

dbt ran their first conference in 2020. 2021 they ran their second, this time it got big.

Phase 2: Growth

MDS BIG WINS

Snowflake IPO late 2020 set things in motion, the biggest software IPO ever was just the beginning of the end for MDS. dbt raised a ton of money, so did Fivetran and most of the others.

EXPLOSION OF MDS PATTERNS / BEST PRACTICES

Locally Optimistic was always and remains an ardent supporter of delivering relatively hype-free and insightful best practices. Their community remains well moderated and insightful, with very friendly and well moderated non-vendor participants. Remains a top place to visit.

Community best practices and advice about starting a company community slack started to pop up. If you were VC backed, you need 1000+ people in your slack otherwise how would you get any product feedback. This worked well in my opinion, despite the snark. If I am legitimately interested in your product I’d love to speak to the engineer building it. Companies started adding the slack activity into their go-to-market strategy, ELT’ing my activity into their CDP for better PLG or whatever.

SMALL MDS CONSULTANCIES

An acquaintance (or quite a few actually) went full throttle on consulting and quickly employed 100s of people and made a ton of revenue with massive companies.2

dbt has a huge list of vendors. This was the most reliable way to make good money in the Modern Data Stack ecosystem, but the allure of building products pulled lots of people into building data tools instead.

YOU ARE NOT DOING MDS / CERTIFICATIONS TITLES

The need to teach analysts how to be Analytics Engineers was pretty clear, they needed git, python, jinja, Data models, star schemas, denormalization, Kimball.

A few courses came out doing this stuff, first was the Analytics Engineers Club. I think the data modelling skills described are super useful, and generally fell into something that Data Engineers didn’t do and Analysts didn’t do, and so they were just ignored.

This wasn’t explicitly gatekeeping, more just demand being met by an increase in supply. I was firmly of the opinion that hiring ops people was beneficial in this space as they could easily grasp the tech and generally had a better sense for the problems that were worth addressing.

MDS VENDORS FUND CONFERENCES

There were a few smaller conferences other than Coalesce (the conference), and Snowflake summits, Data Council seemed to be the most consistently worthwhile.

In 2021 the dbt Coalesce conference peaked:

This cultural energy reaches its crescendo during Coalesce, dbt Labs’ annual conference. The conference ostensibly takes place over a number of live-streamed talks, but its beating heart is on Slack. Every talk inspires a tidal wave of excitement, encouragement, and general good cheer. Every channel is the parents’ section at track meet: Ready to erupt when their kid crosses the finish line, and equally ready to hop the fence and pick them up if they fall. 

MDS ECOSYSTEM INFOGRAPHICS

The vast ecosystem now needed a map, ModernDataStack.XYZ maps out the components, and I think does an adequate job of categorising them.

Here is another map, with their criteria giving you an idea of the qualifying criteria. 3

MDS VENDOR COMPETITION HEATS UP!

Once the ball started rolling, there was a lot of money being poured into the space. New BI tools, new ELT, data quality, data reliability, observability, metrics, semantics, all started being picked over.

I wrote a blog comparing dbt and Dataform. Dataform was acquired as mentioned, and dbt took centre stage, centre diagram.

Two big themes were Reverse ETL and CDP:

Another somewhat-in-the-weeds, but fun to watch part of the landscape has been the tension between Reverse ETL (again, the process of taking data out of the warehouse and putting it back into SaaS and other applications) and Customer Data Platforms (products that aggregate customer data from multiple sources, run analytics on them like segmentation, and enable actions like marketing campaigns). 

Reverse ETL was always a tricky thing. I like the idea of reverse ETL, but ultimately it would often build on very weak foundations:4

The CDP space also sort of heated up, and had significant overlap with MDS. The MDS gave you the tools to construct your own solution, and design the specifics to suit your needs, whereas the CDP kinda gave you much deeper capabilities, but less control (for the customer analytics niche).

FIRST ENTERPRISE VENDOR MDS OFFERINGS

Enterprise oriented MDS style tools and even direct clones started cropping up. The awkwardly named Coalesce was the first early dbt alternative since Dataform (Coalesce is also the name of dbt’s conference). I don’t know much about it.

MDS GARTNER MAGIC QUADRANT

There isn’t a fully fledged Gartner Quadrant. The whole point of MDS was all the unbundled solutions. There was a post on their forum. I predict we will see a “new wave” once all the tools are more consolidated and Gartner figures out how to position all of this.

I guess there are likely enterprise equivalently branded MDS style solutions that have happened. The only funny one I could find within 5 seconds of Googling was IBM Gen z/X.

OFFERINGS TO SOLVE MDS PROBLEMS

I almost wrote a blog decrying the state of things when I was spammed by an ex digitisation evangelist crypto expert, now selling data-something-dot-AI. Selling magic and then supplying junior analysts with git and SQL just stank of an enterprise sales cycle. This was indicative of the beginning of the end.

DON'T DO MDS / ENTERPRISE MDS

Data mesh was the funniest part of the entire data hype cycle. Effectively a fully fledged framework to decentralise data ownership, it read like a Business School Blockchain Certification (apparently, I didn’t read it). The response from the MDS pure snark. Pedram had the last word here. I guess maybe you had to be there.

There was heated posturing on Linkedin about the failure of MDS to properly address the problem of data modelling. Data Vault, Anchor modelling. How about a full circle all the way back to one big table?

This was also an Enterprise / Startup culture clash, but probably mostly a mutual lack of context and incompatible cadence. A 100 person SF startup growing beyond terminal velocity vs a declining mid-tier bank in the Bavarian hinterland aren’t going to meet each other with much common language. I wrote something about that here:

Phase 3: Involution / Stagflation

The cracks started to show in a few ways: cost, complexity, BI tools, sprawl.

Cost:

Snowflake et-al being super expensive was the first ominous crack. When interrogated, massive cloud bills often weren’t attributable to any value. It was presumed when these at kick off that they would be both expensive and lead to value. The expensive part was duly stomached but then the value part was late to the party.

ELT, synonymous with MDS, came under cost pressure. People realised that the work was pretty predictable and so cheaper, better, faster options became abundant.

And then AWS, Salesforce and Snowflake all began to murmur about ‘zero-ETL’ which basically meant they will co-access or whatever the data that is in Salesforce from Snowflake. Basically ruining the Fivetran business model.

“What if we could eliminate ETL entirely? That would be a world we would all love. This is our vision, what we’re calling a zero ETL future.

dbt:

Dbt was really just a victim of success, when you push into the unknown you will find out, and dbt found out. Before dbt, teams kept their reliance on a data warehouse simple and un-collaborative. One engineer would generally be responsible, they would have de facto veto on data modelling changes. dbt democratised that:

Did we achieve more collaboration on an analytics code base? ✅

Did we achieve more leverage through reusable and modular code? ✅

Did we also buy more complexity, resulting in longer maintenance and debugging cycles? Unfortunately, also ✅ 🤓

Turns out the price of enabling people to build a more complex code base is… a more complex codebase, and everything that comes with that.

Solving one problem and in doing so creating another is the essence of progress. I give dbt a pass here. Nonetheless, things got very complex, and not just limited to dbt.

There were numerous hot takes pointing to the shortcomings of dbt, and a slew of dbt alternatives popped up, all “faster horses” in my mind. We need cars.5

dbt having raised a ton of money, had to do layoffs and quickly figure out a business model.

Complexity:

You could be lead to believe that you need one from each of the following “MDS categories”. However most teams generally limited themselves to a BI tool, a database, an ETL tool and dbt.

The issue was the abundance of overlapping options.

Team size:

The other awkward thing to address was the growth in MDS team size. I worked with a team that wanted to double their data team from 20 to 50. MDS became known to run on human middleware:

Cash is injected. This means more employees of all sorts are hired. More employees increases the demand for more reports and analytics. More demand means more human middleware is created in the Clueless layer when you adopt the Modern Data Stack paradigm of throwing everything into your cloud data warehouse of choice.

More Clueless human middleware creates more tables, tables, tables to many more reports and KPIs and metrics. They have to buy new products and hire new people to manage the complexity.

This criticism was semi-reasonable6, but honestly I think this is often just the cycle of technology. Something comes along, presents an opportunity to differentiate, it works for some, it doesn’t work for others. People need to be involved. How many people? Probably a few. Oops too many. OK less people.

A more primary issue with buying anything is knowing if you are ready for it. Does this company need a better data capability? Falling behind is a real risk, that compounds!

The Data Stack was sold broadly, for some it worked, quite often it didn’t.

BI tools:

Business Intelligence tools continued to underwhelm, primarily because of the split between traditional reporting and exploratory analytics.

The paradigm of “reporting” is in my mind a dead end. Like delivering a menu and then never taking an order, BI tools were informative instead of interactive.

Traditional BI just don’t move the needle in the same way that newer tools like Hex do. (Hex described this paradigm in deleted article, they now position themselves as a data tool that does reporting too). I use Hex daily. It is relatively cheap, it works very well and has sufficient depth to replace a fair chunk of MDS and technology infrastructure too.

To be fair to BI tools, they were the last mile delivery problem built on a relative house of cards, so were pretty much destined to be the pain cafe.

Sprawl

There was Vocal Criticism from product analytics people about the sprawling nature of MDS.

Product analytics is a mature side-car to the MDS, and the tooling built by Posthog et al is generally end-to-end integrated. From their perspective, the MDS approach led to poor outcomes:7

Those are a few of the criticisms that illustrate the point. Essentially the belleweather for the change in direction of the flock.

MDS IS DEAD POSTS

There were a few. This became a bit of a trope. Meta analysis of the trope is far more palatable. Hope you agree.

BACK TO BASICS MOVEMENT

There was always talk of a bundling. In essence the MDS was unbundling, and at some stage the tide would turn.

This was actually discussed at length in early 2022, where most everyone agreed that the unbundled approach was great for many things (experimentation, investing, entertainment, curiosity), but it wasn’t very productive.

MDS LATE ADOPTERS

There is still the MDSFest:

“A community-led celebration of ideas and perspectives on the modern data stack”.

Sounds great but I literally just came across it researching for this blog so I don’t actually know much about it

I don’t know where this video comes from but I think it demonstrates the idea that one person can pull together an entirely viable, semi-scalable data platform from the best of the open-source stuff, as was always intended. Silver lining.

EVANGELISTS MOURN STATE OF WAY

Some of the early evangelists mourned their loss. It did literally feel magic. You could achieve so much with so little. This was a reality.

A shrinking pie makes zero sum games more likely too. A shrinking pie means you have to work pretty hard just to stem the flow:

Everyone Wants a Piece of the Pie, Nobody Wants to Bake
…
if you don’t build things to solve a pain you’ve had in the hopes that it’ll solve someone else’s, if you don’t give away your hard work for free, then kindly, please, shut the fuck up.

Matt Turk knows all about all of this, having documented the data space for ages:

The MDS is now under pressure. In a world of tight budgets and rationalization, it is almost too obvious a target. It’s complex (as customers need to stitch everything together and deal with multiple vendors). It’s expensive (lots of copying and moving data; every vendor in the chain wants their revenue and margin; customers often need an in-house team of data engineers to make it all work, etc). And it is, arguably, elitist (as those are the most bleeding-edge, best-in-breed tools, serving the needs of the more sophisticated users with the more advanced use cases).

Overfunded startups, overcrowded teams, too many compute credits, too much badly structured SQL and lots of criticism.

The substance boiled down:

MDS was a series of relatively experimental tools strung together to demonstrate varyingly good levels of Product Market Fit, but not quite demonstrating an ideal operating model .

The result is a great target for more refined, more niche, bundled equivalents.

Phase 4: Postcycle

END OF MDS STATUS GAMES

There is less attention and so less opportunity for status games. Most people have some trusted source of information and they rely on it, less interested in the minutia and more worried with whatever their pressing problems are.

A large chunk of data practitioners are now product focussed, in role, in company, in orientation. This is sensible as product is probably one of the best maturation directions, as it was a primary consumer of data outputs. ex-Data know how far to trust the data systems and what lies they tell. The same applies to operations.

MDS CONSOLIDATES WITH FOCUS

The data systems are largely good enough and so the bottleneck becomes what to do with the data. Data engineering was the bottleneck. In January 2022 I gave the opinion that Data Engineering was no longer the primary bottleneck to delivering insights/value/whatever.

Now in August 2023, I will now say that the Data Function is no longer the primary bottleneck in delivering insights/value. A good data team can ingest, model, analyse and distribute insights pretty easily. The challenges are now more subtle:8

Track 1: “All I want is to know what's different”

Track 2: “The emotionally informed company”

Track 3: “The truth is out there The only thing stopping us from finding it is us”

Track 4: “Will we ever have clean data? Probably not, but maybe we can work with messy data”

Matt Turk describes where things go from here:

The convergence of streaming and batch processing is an evergreen, and important theme. So is the convergence of transactional (OLTP) and analytical (OLAP) workloads

This Analytics vs Operational thing is critical. All I will add is that all data problems stem from the fact that the blue and a pink blob are handled by different teams.

MDS LEFT US WITH MORE / PRACTITIONERS STILL CARE

The whole argument around MDS now mostly dusty, it can simply be described as a good idea that explored all the avenues and turned over all the stones.

Data technology was just a nightmare before, and a better solution is always necessary. The unbundling of the big systems into discrete elements was effective, and allowed people to experiment, learn, share and iterate on that cycle towards something that was very effective. Timo Dechau describes this:

The funny thing about evolution and potentially the one often missed out. Evolution is never linear. It branches out, explores, and creates massive amounts of variants. That is the beauty of it.

But it is also why there is never “the” next. But hundreds of next. And out of them, at some point, we will see a step changing the ways in such a good way that we could declare it as a new paradigm.

We are not there yet. But we can already see the branches, which is exciting.

He has another good line on of thinking that maybe Product and MDS will converge.

Product analytics on top of your events in your data warehouse. No more weird data loadings and enrichment (where most of them never worked). And mostly, no two setups for classic BI and product analytics use cases. We spent some more time with this approach later.

IN CLOSING

STILL REAL PROBLEMS TO BE SOLVED

The opportunities are there. My core issue with MDS was that data modelling remained a complete nightmare. As the SaaS systems that run a business got more complex, the effort to consolidate went up and the accuracy of the consolidation went down.

My take on data modelling:

1. There is an optimum level of data modelling done by software developers building apps and not just leaving it to the data team (Analytics vs Operational)

   

2. There is more work to be done exposing better data models via API from SaaS companies, and especially internally.

3. There absolutely has to be better (easier, pre-populated, more automated, less fragile, less complicated) data modelling techniques.

The immediate future for data modelling involves an important role in the next big thing

Enterprise data systems can often be quite distinct from their smaller startup cousins. This post takes a look at how our conversations around data systems and techniques need to be more sensitive to context, specifically when conversationalists have varying backgrounds.

A puddle-deep dive into the world of enterprises and how they compare to startups.

Does it generalise?

A key thing I come back to with working it data, writing about it and especially reading about it, is the following: does this generalise? Does this apply to a general context?

We all have a specific set of experiences and sufficient time to learn from others. This gives each of us the perspective that forms our opinions on the world.

So occasionally, we will say something like1:

Matt Arderne @mattarderne

I think tech/SW teams should (be required) to build APIs of core business data concepts (customers, orders, products) way earlier The anti-pattern of this being defined by a new (data) team in a new tech paradigm (dbt) and then exposed in a new tool (BI) causes much pain

2:34 PM ∙ Nov 17, 2022

---

39Likes2Retweets

This tweet led to fascinating conversations with many people, experts and non-experts alike.

What is even more fascinating, is that the context that caused this statement to resonate so strongly is entirely presumed, or even unknown.2 The context, the specific experiences that led to the specific insight, is unknown.

This I would argue is a key issue with a few of the rifts in data discussions. Data mesh, modern data stack, bundling, data modelling, dbt, human middleware, contracts, purpose, gods, etcetera.3

How many of these rifts are due to a lack of generalised context?

Let’s have a look.

Who are we?

Data people do come from a pretty consistent set of contexts:4

The monolithic startup driven by a charismatic CEO. The meshy enterprise and its tangled mess of systems. The methodical R&D group with its tinkering innovators. The mutinous midsize company and its C-Suite fights over whose department initiatives are really driving growth.

Across all of these complexity busting teams, possibly the clearest dimension that I've noticed is the different, almost opposed operating models between the scrappy startup/scaleup mode and the sprawling enterprise mode when it comes to implementing data solutions and systems.

Here are a few key ways of describing characteristics that really distinguish:

• CTO vs CIO5 responsible for data

• Data needs to be useful vs data needs to be accurate

• Growing vs stable (dying)

• Emergent vs traditional

• Product market fit vs risk averse

This is worth discussing because we have long since reached the point where these conversations feel quite like worlds colliding

(the enterprise data-mesh car crash through the fence of startup analytics world),

and this was like two previously un-contacted tribes with each their own gods and deities coming together and not loving the look of each other. People just screaming right past each other. Here is me screaming:

Matt Arderne @mattarderne

@mullinsms @DSJayatillake @DomenicRavita @getdbt @fivetran This circular conversation seems most common when the enterprise context clashes with rapid growth context. Decay vs Acceleration orientation largely not reconcilable, which IMO explain these

8:04 AM ∙ Oct 13, 2022

My Background

I have always worked in and with smaller businesses, the median people number being ~150. I briefly worked as a solutions engineer for a company that nearly exclusively sold products to big enterprise companies, multinationals, banks.

This experience was not agreeable and shocked me in a few ways. Primarily, how incredibly complicated the enterprise customer operations can be, and how dysfunctionally the enterprise customers ran their technology projects.

Complicated is fine, complicated is not inherently complex, and just requires careful consideration, rather than rocket science, to figure out what to do. But in these older businesses, what to do is staggeringly complicated.

And so if you have worked in a startup, you may experience enterprise as static, as if nothing happens.

Well, things happen, just nothing changes,

At least not satisfyingly so, and certainly not quick enough for any feedback loops to develop, and absolutely not long enough for anyone to be around for long enough to be held accountable for their actions.

What ends up happening is pretty organic, often incredibly misguided, and very likely even counterproductive without realising it:

Not only do systems expand well beyond their original goals, but as they evolve they tend to oppose even their own original goals. … For example, incentive reward systems set up in business can have the effect of institutionalizing mediocrity. This leads to the following principle. Systems tend to oppose their own proper function.

Working in a company selling software was great, but selling to such established mega-corps was not my style of toasted cheese sandwich.

This blog has simmered as a comparison between Startup-land and "your typical lower tier enterprise". The result is a massive generalisation, highlighting the best of startups and comparing them to the worst of enterprise. I have a minor sliver of context, and I'm adding it to the mix. I'm not an enterprise/startup expert, but I do have experience, which means I have some context to share.

Many people in this space go their whole lives not looking under the hood of an enterprise, or startup, and this is for them.

So what about startups

Anyone reading this probably has an idea of startup land.

Startups have the benefit of a mostly clean slate, minimal baggage and an attitude of must take the time to get the tech systems right. Data is very often part of the strategic differentiation (guided or misguided as it may be).

Single data system, single tech, migrate quickly, do it now, cutting edge, sorry cancel the contract we turned that off.

Often wrong but always correcting, both the data stack and the company. Startups pivot on a cent(dime/penny), making extremely confident moves, with everyone in the team ready to have their role/responsibility severely disrupted, and be thrilled about it (unless you get fired).

Some great points I seem to have copied from a slack about what to expect at a startup if you’ve only worked in an enterprise:

You will ALWAYS be building the plane as you fly it. Take time to step back and really think about your foundation and the scale it needs to operate at.

My advice would be to try to be as ‘T’ shaped as possible. Don’t worry about being amazing at lots of things - just be good enough at lots of things and best-in-team at one thing in particular.

Just do things, permission is unaffordable overhead.

Prioritize aggressively because if you do the wrong things the company dies.

Hot on the trail of customer needs and wants, iterating through the early stages of getting product market fit, and then once the scaleup stage is reached, all hands to the optimisation engine.

Startups are relatively simple - everyone is aligned. If not then it is probably just a small enterprise.

Summary

• Uncompromised, as everything matters, and the business will probably fail

• Any single failure could be fatal, but only can be minimally mitigated

• Aligned incentives

• Fragile, like a newly lit fire with massive potential 🧨

Scale-ups?

Scale-ups are what happens when a startup does exceedingly well (product market fit or excessive funding) and transitions from chaos into chaos on a mad hiring spree. The metamorphosis is described here:

The growth of a company from startup to enterprise has a lot in common with the growth of a caterpillar into a butterfly. And those parallels are especially meaningful when you’re in the goopy, amorphous pupa phase.

It can be disorienting to go from a stage where the emphasis is on quick execution for immediate results to a stage where ideas take longer to incubate and impact is measured in years, not days.

Often enough a chaotic nightmare where no one knows what truly matters. The inherent value of this stage is being tested by the current lack of growth VC funding.

Pouring fuel on the fire now that you won’t smother it 🔥

So then what about enterprise

Enterprise is entirely different. With endless silos, centralising and decentralising efforts, complicated by mergers, locations, borders, timezones and subdivisions. The focus of data is on basic coherence, control, and standardisation.

I'm not talking about Apple here. FAANG doesn't have this issue, they just build solutions. CV buffering for engineers, they have other issues. They have issues relating to hiring people they don't need to prevent other FAANGs from having access to them, and other rumours about deliberately depressing profits.6

I am talking about your mid-tier banks, in forgotten regions. Companies that need technology to survive, but have a complicated relationship with it. Think COBOL stories. Think of General Electric post-merger with one of the 100's of companies in one of the very many sectors. Declining profits, fired CEOs, disillusioned staff. Failed ERP implementations. The long tail of companies that just exist on momentum and middle management careering.

At a true enterprise, one can expect many legacy tech stacks, each with a plan and migration timeline. Keep it reliable, supported, staffed and contracted, ideally a single vendor.

What this leads to is Zombie Projects, with incredibly disillusioned teams running them:

when a project has failed or has ballooned in size to the extent that it will never be completed, and everyone knows it. When that happens, and no one wants to face the facts, and so the project continues to move forward, then it becomes a BS project.

Individuals start to realise that the project has died but will remain funded, and the system supports itself. They realise that despite whether they work or not, nothing happens and nothing matters, other than the charade itself. A reminder that this really drains motivation.

That intro is probably a bit harsh, What does the enterprise do well?
Running incredibly complex, massively scaled businesses, often successfully, credit where it is due.

Randall Koutnik @rkoutnik

Working at an early startup is like driving a go-kart. Feels fast only because it's small, you're not actually going quickly. Working at a mature company is like riding a plane. Doesn't feel like you're moving at all until you look out the window and have gone 100s of miles

11:58 PM ∙ Mar 8, 2022

---

21Likes2Retweets

Hero Warship

Another recognisable enterprise pattern is hiring saviours.

Person X, accolade Y, education Z (MBA) is heralded as the big new hire with the big new plan, assigned a big new budget to (finally) achieve the big broad goal.

Hero comes off the back of recent success in the exact same situation at a competitor. They start with an investigation into the “business needs” through many committees, leading to a feature hit-list-box-ticking exercise. Eventually, only massive companies like Oracle or IBM are in the running.

Cynically, this project is entirely for the purpose of leveraging it into a new role at a new bank (Failing Up), years before the project has any chance of fruition.

Another version of this game: In walk the management consultants. Strategise the flavour of the month (Modern Data Mesh), sell the dream, plan the diagram:

optimise cut reduce expand single source modern truth AI prediction profit, drops mic

The contract staffing company arrives, waterfall chart in hand and by the time the paint has dried most of the original stakeholders have jumped ship as digitisation data AI blockchain cost management experts, and are spamming the success of the project before it has even begun!

What are you sinking about

I think of enterprise as a system that has built so many incredibly reliable systems to prevent the business from failing that they become rigid.

Incredibly strong. Incredibly inflexible. Entire functional areas begin to decay, while the rigidity remains.

Put people in charge of rules, meetings, and forms, and their first inkling will be “there should be more rules, meetings, and forms”.

The rigidity and strength prevent flexibility, and so change becomes increasingly unlikely. This eventually leads to a state of dysfunction. The structure remains, but the function gradually dissolves. Idiosyncrasies like change review boards outright inhibiting changes.

Broken systems, operating in an organic state of failure:

The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe.

Summary

• Compromised, primarily due to risk aversion

• Severe misalignment

• Hot embers, of a big fire 📛

Looking Ahead

As we have seen, the enterprise has largely taken note of the successes (and failings) of the organic brew of recent data tech.

My guess is that over the next few years, we will see this accelerate (they don’t read the salty tweets), but it will largely squish the pulp out of it in the outsourced and cross-matrix-managed divisional regional organisational chaos, and after decades of implementation, will abandon it to the pile of supported and business-critical but largely ignored technology.

Alongside we’ll see ever more of the offering of cargo-cult as a service from the consultancies forcing their way into the fray, with AI names and outsourced quasi-bundled solutions.

Often these kinds of trends are misguided, but some are indeed necessary. Digitisation? Yes if you must, no one wants to call for a delivery update. Cloud? Probably.

So Modern Data Stack then? Well, it comes with a way of working, an ethos around software best practices, some thoughts around who is responsible for what, and how to engage as an impactful product minded team.

Therein lies hope. Where Hadoop was a big data solution, Modern analytics is possibly a process improvement. Doing things a bit more coherently. A slightly easier way to do the same old.

But I worry that it is probably not enough, or not even the case.

The ease with which enterprise data modelling experts have co-opted the data modelling conversation on LinkedIn tells me that the more likely post-fact analysis is going to lead to something along the lines of:

MDS brought Enterprise capabilities to Startups, and not the other way around.

That is a shuddering thought if there ever was one?

As mentioned, and regardless, Modern Data Stack has started simmering in the Enterprise, and is now well on its way. A new song with the same dance.

So what?

Back to the opening point on context.

Take a moment to understand the context behind opinions, product features, books, blogs, tweets and toots.

When consuming in this space it is critical to ensure the relevance of what you are reading, to you.

Is that (this) advice or content applicable to you?

More importantly, the person writing that (this) blog. What is their background? What are they saying. What have they not said?

Thanks to Simon Späti for insightful feedback on an early draft. Simon published a very insightful analysis on how enterprise data systems actually work. Also thanks to my team of editors for the proofreading.

20 Feb 2023 edit - Google is indeed an enterprise by the definition broadly described in this post

No one surfs anymore, it’s too crowded

September arrives, and coincidence or not, the first week of Autumn brings with it two long-period hurricane swells, colder days, rain, frontal weather and a few days of brisk Northerly winds. The changing of the seasons in the Northern Hemisphere can be pretty punctual, as if a queue of storms had already formed, waiting ready for departure (something to do with the jet stream). Departing in an orderly fashion, right on time - one two three straight into the coast.

Their arrival, much predicted by the breathless call over the premium online emergency global wave alert system, much like the Hawaiian missile alert system, is prone to false positives.

Nonetheless, the surfers of London heed the call, firing up the chic-camper-conversions, having phoned up ahead to book a soft board for 2.5hrs, or more likely ordered a new hybrid twin-pin with same-day delivery.

With Flo in tow, swimming wetsuits and dry robes in the back for the 4/5hr Friday burn “down country” (down must be the opposite of up, as in “up country” - the other direction meaning London).

And so set out the Dry Robe Wankers, the pejorative term for the influx, and the sales of camouflage having dropped since this made the news (I don’t own a robe, but I did live in London. Conflicted).(Update: I now own a robe, best ever!)

Intermediates Only Please, this is Britain

Saturday 3rd September arrives, with it a stern warning on the global alert “beginners stick to sheltered bays”. The rule abiders abide, strictly. Surfing, hilariously, has no UK commissioner, no council and apparently no regulatory body, which is quite literally unheard of in the UK hobby scene. There is the BSA (I guess the acronym) that run the surf contests, but there is no RYA (Royal Yachting Association) equivalent, to hand out “intermediate” badges. And so the majority remains in a state of purgatory, formally a nation of beginners.

The swell arrives, and the high tide shelters formed by the bays provide some much-needed orientation for the hordes. A natural queue is formed, and beyond it lies a wild ocean. The wildest ocean I’d seen in months.

The irony is that the reason I was in attendance alongside the horde was purely to find some sheltered waves to continue my foil journey. As I indicated previously, I am somewhat hooked, and thus need a place to practice my art. The summer months lead me to exposed beach breaks, honestly occasionally wishing that I had my 6’6 mid-life-mid-length instead of the bloody foil.

For the majority of this time, the surf has been perfect, absolutely perfect for a beginner. Not to mention the water has been incredibly warm (hot in a 4/3mm), and endless sunny days (also in duration, 6am to 9pm) meant that for the bulk of the summer, double sessions were viable. Adding to this the foiling game of pumping out to link 2 or 3 waves at a time means that the annual wave count has certainly started to rip up away from the 1 year moving average.

This abundance of knee-high ripplers went unchallenged, and the only people that joined us were a euro couple who even went so far as to comment to our crew on the beach that we were the friendliest and most accommodating foilers they had encountered. How many foilers and how many surf sessions they could possibly have had to form this opinion I don’t know, but they both were taking off on green waves and heading down the line, along our perfectly formed peak that faded into a nice big rip which ripped us right back out to sea.

The bay of plenty

Back to The Big Saturday, and the bay was jam-packed. Ominous signs forming in the typically deserted 7am car park.

Many, many, campervans, and quite a few with surf kayaks emerging.

The parking lot is usually rather sparse at this time, weekend or not, but today was quite a scene. Immediate reservations around adding a foil to the mix, but a quick look at the water satisfies me that if I’m quick, the session won’t last that long and I’ll be in and out before the bulk boat up.

Summerleaze bay in Bude is a relatively protected bay at mid tide upwards, with a seawall providing a barrier for boats and thus a natural haven from the elements for surfers too. Coupled with a very high tide, it takes all the energy out of swells and makes for a very peaceful sleepy running wave.

My arrival was carefully timed to match the tide, and as the tide had topped out and was starting to ebb, I had about an hour or two before the tide starts to pinch in the mouth of the bay, between Barrel - a green barrel channel marker, very useful navigation aid marking the Starboard channel boundary, and the relatively un-documented Cross Rock - an iron cross embedded into a rock, below the water level at any boat-approachable tide, and rather deadly looking. (Update: apparently it marks mid-tide)

On the foil we can use the boat entrance as a channel to pump the foil back out to catch another wave. This “pump” is the main cause of derision towards foilers. “Stupid things why do they bounce around so much”.

This is the exact same mechanism that surfers employ to generate speed, in surfing known as the Hungtington Hop (not sure I agree with the physics prof on why it works, but then I’m not a prof). In surfing, this is applied to stay on the wave, but usually only when the waves are especially gutless.

foiling is pumping - you’ve got to pump (it up)

With foiling, the same technique is also applied, but can lead to significant advantages as it allows you to do two things: 1. accelerate down the wave, 2. accelerate beyond the wave, carrying the momentum and then continue adding to it, allowing you to pump out and catch another wave as part of the same takeoff. This “link” is the first hurdle of foiling. Much like doing pull-ups, the first hurdle to foiling quickly becomes a physical endurance repetition game.

How quickly can I get off this wave and pump back out to get another, this weighed against riding the wave to the end and having to pump further back out for another.

In 2018 Kai Lenny managed 11 waves in a row with what would today be regarded as a ridiculously inefficient way to pump for only one wave, all the credit to Kai who is quite the human-machine. An antipodal man recently went on to smash the record by catching a wave and continuously pumping between waves for 2.5 hours.

Catching 4 waves leaves me wishing I’d rather hit Cross Rock, and so the recent record clearly indicates that the technique involved transforms the pump from anaerobic to aerobic, where the body can maintain lactate levels etcetera!

So the pumping is a big part of the thing. Pump pump glide, pump pump glide.

The idea is that once you can pump effectively and reliably, you can catch as many waves as you please. The other attraction is pumping out away from the crowd, as you only need a breaking wave to takeoff on, and once you are up, you can very reliably head out to sea at about 15kph, to catch ocean swells long before they break.

Another attraction is “laddering” up a set, where you pump out to sea, and on the way out catch a short break by catching a wave, getting some speed on each wave or second wave of the set, giving yourself a quick rest and a speed boost. Interestingly, the faster you go, the less effort is required to maintain the speed (caveats etc), and also when you catch a wave, you go fastest down the line. This is the same as with surfing, but the acceleration and top speed that a wave provides to the foil is quite incredible, and the same physics that applies to squeezing a olive/watermelon seed to shoot it applies here - you get squeezed and shot down the line, taking that speed with you as you quickly dip in and out of waves to keep the momentum up and the heart-rate down.

Summerleaze provides a very gutless wave, so much so that traditional short-board surfers mostly ignore it. We foilers can takeoff on a small breaking bump, and as it fades out (so the surfers generally ignore these), we ride the swell through the deep water channel and pump back along the breakwater channel, boosted by the very strong rip that forms, out to catch another wave in the set. The mouth of the bay at high tide provides a natural sand bar that causes the waves to pitch up, making the link easier and more attainable, but still not of interest to the surf crew. Riding this wave back along the channel, making a few cutbacks to stay near the “pocket” of the wave.

The foil feels like an oversensitive highly calibrated instrument and amplifies every single impulse of the sea. A bump needs to be navigated, double-up waves create a half-pipe, turbulence from a broken wave is literally like flying through a massive storm, hoping that just holding tight will get you through it. Pumping back out to sea, swells provide an opportunity to pump up and down like a skateboard ramp, further accelerating if you get it right and stopping you short if you get it wrong.

Foils also have a critical altitude, above which the foil “breaches” the water and you leave the board as if it hit a pothole or stone on a skateboard, an instant stop as the wing loses lift. These are avoided by keeping lower on the mast, but the more of the mast that is in the water, the less efficiently it rides as you have the mast not generating lift but generating significant drag. A delicate balance as you navigate the bumps and troughs of the sea, trying to eke out energy from the ocean and conserve it.

Combining these aspects means you have set yourself up with an invisible snakes and ladders obstacle course, a literally multi-dimensional game of strategy and endurance to try and get as far and as many as you can. Ideally, you’d catch a smaller wave before a set arrives, timing it so that you arrived back at the main peak to coincide with a set wave, and then laddering through the set to find the biggest lump. Riding that wave through to the inside carving as many or as few turns as needed and then pumping back to the takeoff for a rest. No surfers are at all interested in the same waves, and so the bay turns into something of a skateboard pump track, but without constraints and only your fitness and ability preventing you from riding an endless flowing wave.

Our ideal wave is one that breaks and then fades. This is optimal for a few reasons. The fade means that it is running into deeper water and so our 75cm+ draft won’t run aground. We need a wave to break as the boards are generally smaller and stubbier, to maximise pump efficiency. I have never been as finely tuned into a barely breaking wave and whether I can catch it with my 32 litre 4’4 board. Strong shoulders and a tired back. These “chip-in” waves are potentially the only part of the experience that has any interference with the regular surf crew, and we try and minimise them as much as possible.

Kayaks. KAYAKS!

As the session progressed, and the boats started to arrive, it became clear we had a slight clash of wavelength. For some reason that I do not yet understand, they seemed to be partial to the same waves that we needed to chip in. The boats in question are a novel creation, something that I had literally never encountered in my 20 and a few years of ocean interest. They are formally surf kayaks, though you’d still be puzzled clicking on that link as to which variant. The 11th resulting image is the answer.

It is a sit-in wave-ski type boat (right below), which I think primarily stems from the river kayaking scene in the UK (riverless SA might explain the African drought of these). Apparently, they are easier than the traditional wave-ski so common in SA in the 80s (or MacSki, left below) as the centre of gravity is lower, and they are generally a bit longer. They have a splash hood and quite a smooth long rocker line and seem to be generally quite a bit easier to throw around, and also paddle more easily, with a few of them getting into waves surprisingly early, with much less wild windmilling.

This early entry also meant that they could glide through the slow parts of the wave, linking the outside section into the inside, typically the domain of the foil. And there were lots of them. An incredible amount, an unnatural amount. It turns out that the world champs are being held in Bude in 2022, and this appeared to be the England crew, hunting in packs as it seems, turning up for some team building and last-minute coaching. As fellow marginalised surf craft, we discussed rockers and foils while trying to keep out of each others way. I say marginalised because that is the category, distinct from “alternative”, which is far more fashionable and palatable for the core surf fashionista, and typically is approved only because it can catch less waves than a surfer.

Alternative is shorthand for bodysurf, Alaia, surf-mat.

Marginalised are the SUP, kayak, and most certainly the foil brigade, and probably for good reason.

Through all of this, the tide started to drain and the bay became smaller, the rip became more pronounced, and the session became slightly more location-condensed. The double takeoff spot started to merge into one, and the fishing-boat channel became more narrow, and also started to become the shoulder to the main peak when larger sets arrived. The effect was that the foil could no longer comfortably ride the rip out as it became crowded, and the sets were pushing through with too much energy to really ride comfortably. It was at this point that I realised I would be no longer a sensible participant in this session, but hung around for another 30min to catch a “wave in”.

Beginners take care

The observation primarily in this period of time was around how the median competence level was somewhat lower than I had realised. This is an evolving observation and will require some baseline comparison when I’m back in SA, SA being the baseline, specifically a few spots around that coast that have a similar attraction to beginner/intermediate surfers. Excluding the surf boats for now as they are more of an oddity than anything else.

Surfing in the UK is incredibly popular, but as I discovered on my recent Bude Surf Life Saving club induction, “do you surf” equates in most people’s ears to “have you ever tried to surf”, with a confident Yes being the answer.

Much like my answer any time, erring on the side of blinding overconfidence lest it cost me an opportunity for fun; “yes I can certainly ride/drive/operate/command that bike/plane/digger/fleet”

Thus the whole country in effect surfs, as it is a seemingly compulsory part of the summer beach kit to add a £35 supermarket-bought wetsuit to the kit, along with the beach windscreen. Both peculiarities I have not encountered in SA. The wetsuits potentially because there are too few surfers in SA to make a commodity wetsuit viable for retail, and the windscreen because the SA sun is too intense to prioritise a windscreen over a beach umbrella.

In fairness I think the screen is great, as it reduces the likelihood of the umbrella blowing away, and reduces the wind-chill factor, which in the UK is quite a factor indeed for the marginal summer days. I wonder if they would be a hit in SA if offered.

And so the observation continued, as a BIG set rolled in, now hitting the only peak square on, the channel (or shoulder), which was now indeed shoulder to shoulder, full of boats and the more competent surfers taking the rip back out, and the inside section a chaotic mix of intermediates bailing their boards to the chest high set (that’s 1-2ft South African, don’t @me). I blame The Wave, as the world’s easiest scapegoat for ruining surfing, primarily because they have created a national benchmark, and stamped a label on the Intermediate:

Surfers who consistently ride white water waves and are able to catch an unbroken wave The intermediate surf session is open to everyone aged 6+ who are able to swim 25m unaided and are confident they have the required skills to ride this wave

Swim 25m unaided! Well there you have it, a lower bar I cannot fathom.

Intermediates at The Wave (capitalising in case it isn’t clear the link is to THE W.A.V.E!) have potentially equated their Wave ripping abilities in the knee-high chlorine mush as a global accreditation, and also a verifiable licence to ignore the MagicSurfLineWeed advice that applies to beginners - “beginners stick to sheltered bays/wait this one out”.

What they do teach you at The Wave is that if you keep f&*king up The Wave for everyone else, then you get kicked out of the water and sent to an easier session. Having spent £60 for an hour session to watch as Advanced™️ surfers attempt the Expert takeoff and blow it 7 times, I must admit I would love Love LOVE for The Wave to more aggressively police this, maybe printing NFT badges for those that don’t ruin the session and allow them to level up rather than the self accreditation system.

What they don’t teach at The Wave is how to spot a cleanup set that has quietly coaxed all the Interbeginners into the inside. These cleanup sets now draining neatly off a more pronounced sandbank with few fewer meters of waver over it, and starting to peel into the shoulder. Subtle carnage ensues as every single person ditches their board regardless of the chances of pushing through this now chest high wave, boards rattling and bashing as the crew untangles and flops towards another wave.

Clogged Arteries

The overwhelming feeling through all of this is that there are too many people for any one individual to progress.

Progression is a combination of repetition in a valid environment and a feedback loop. Coaches abound in the busy lineup inside, but the beginner cannot make the leap to intermediate (and I mean here the global intermediate - can surf varying conditions, get into position reliably and reliably catch most waves, and most importantly, not cause grief to another surfer).

There is a certain limit to the number of waves that can be surfed per day. Assuming around 4 to 5 surfers can reasonably surf each wave (at true beginner level), and there are 4-5 waves in a set, that means 20 surfers can at best get a wave in each set, at the most generous. 5 surfers surfing a wave doesn’t lead to much progression other than avoiding bashing (or not) into each other.

This is all a very roundabout way of saying that a Covid-driven growth of surfing due to any number of interacting elements has led to an overshoot, whereby the component surfers of the surfing mass are starving each other of waves, clamouring for some photosynthetic light that feeds growth (waves), and in the clamour are preventing growth. A Malthusian reading on this would say that surfing has overshot the carrying capacity, and I suppose the evidence was in favour, as I would suggest that the perfect spot on a perfect day was just far too crowded.

This is not to say that the ocean is full, but the commodified surfing experience has potentially been tapped out, the objective becoming self-defeating. To learn to surf today must be the easiest and hardest thing in history. Easiest because there is nothing undocumented, nothing that can’t be coached 1,2,3 (some things money can buy), and hardest because there is very little chance of progression in the quagmire of the beginner surf zone. Waves have a very inelastic supply, no matter the demand.

I often wonder if this is what causes the UK to have a markedly different feeling in the water to what I’m familiar with in SA. The UK crowd seems slightly more dense, slightly less competent and there is a significant breakdown at the intermediate level, with far too few people aware of what is going on, leading to more drop-ins, more bailed boards, and far more close calls in an otherwise perfectly viable lineup than I have ever experienced.

Perhaps this is more due to my foiling perspective. It is like living at toddler height again, taking on an old perspective on familiar terrain and realising the make-up of the ecosystem has changed. Perhaps in SA, the lineups are equally clogged, and it is only because I’m foiling on the periphery of beginner waves that this has become apparent to me.

Like with all things, people lose interest and move on, this will be the likely outcome like it was with sailing, windsurfing, kitesurfing, and especially Rollerblading. Without some or other invention, the carrying capacity has been hit. I imagine it felt the same way for any surfer of a previous generation as the lineups became more crowded, and no doubt the true carrying capacity was always beyond the current generation’s imagination to believe.

Foil Therapy

This brings me back to my foil, and its ability to artificially increase the carrying capacity of this humble bay in North Cornwall. Suddenly the spot became alive with an entirely new dimension, untapped waves that were going un-ridden. We early brave pioneers who break out of the blinkered obsession with the last gasp of a wave ending its monumental trans-Atlantic journey, looking out to sea and seeing the lines for what they truly are, opportunities to be surfed, and not just opportunities for a brief burst of fleeting joy over a sufficiently shallow sandbank.

I probably should exclude Me from that We. I participate as a likely candidate for the “something new” brigade. Competent at everything, excellent at nothing, never an expert. Experience does not cause expertise, lest someone insist that they’ve been surfing for Xteen years and consider themselves something of an expert, myself included, despite my 3 sessions on expert setting at The Wave.

Foiling attracts those who can’t help themselves but try something new, and it proves the point that all the surf foilers I know are varyingly ex-kiters/windsurfers (and naturally lifelong surfers) who got bored and decided they needed to learn something new.

So who are foilers? They are addicts. Addicts to the progression curve. Flow state demons. What usually happens is the challenges become too high and the experience falls to boredom.

Ask any foiler if they can kitesurf, wing, foil, surf, hang-glide, fly and wakeboard. Yes.

Ask them if they are world-class at anything. No.

Expert generalists are a unifying theme. I surprised everyone at a work office party when there was a cable-ski on hand and threw some backrolls, later surprising no-one. “First Try Arderne” at the climbing gym, giving away the expectation.

And as I called out in the previous post on foiling, I’ve actively avoided getting involved, waiting until such time as the foil was sufficiently developed and enhanced to a degree that was ready for my growth oriented interest, with a range of brands, teams and disciplines. It is no surprise that I have joined as the ground swell is strongest.

It amounts to the same thing in the end - surfers who can spot an opportunity for a new thrill, and it is certainly thrilling to ride big open swells, as much fun as it is to rip down otherwise unremarkable waves at incredible speeds, and carry this momentum out the wave and into the otherwise flat sea, to link into the next surge in energy, in a true and continuous flow state.

Appendix:

It is indeed rather crowded

This is the most astounding little sequence, from a surf-spot that I have spent months surfing, to see this is almost beyond comprehension, and yet here we have it, an illustration of the point.

“When I told them the safest would be that they leave paddling through the channel as it was dangerous for them, he looked at me and answered with an eastern european accent “We know what we are doing”.”

Clearly.

Source with details

South Africa is possibly less crowded

Big bay, a popular city surf break in the city of Cape Town with a larger population, similar if slightly colder water, many more sharks and similarly mediocre waves, seemed to have generally fewer surfers, and possibly unusually, a higher ratio of female to male surfers.

(not scientific, I was chasing seagulls)

Where am I?

Throughout history, this question has usually been resolved somewhat approximately, but always with a strong desire for better accuracy.

From the perspective of explorers deep in the fog of discovering new worlds, each incremental step in increasingly accurate methods of navigation was a new opportunity, much like an entrepreneur sees the incremental tech capabilities as a wonder of possibilities.

Let us pick up the thread of increasing accuracy at the very interesting inflection point of celestial navigation.

Using the angle of stars to the horizon, coupled with some maths and reference charts, one can approximate latitude, with fair accuracy.

Latitude (remember, steps on a La(t)dder) are the ones that go across the map. How far North or South.

Longitude (the down ones), however could only be very roughly estimated due to the lack of effective seagoing clocks and the process being more complicated.

For most of history, Marine Navigators could at best get to the latitude they knew their destination was on, and sail along along that latitude until they bumped into their destination (Land Ho!), or whatever was between them and the destination.

Often this was something hard and sharp.

Crossing the Atlantic Ocean between Europe and the Caribbean (see figure 1) was pretty attainable, with a great heuristic:

Sail South until the butter melts, and then follow the Sun.

The butter melting is the approximate Latitude that you are aiming for in the Caribbean. Following the Sun is heading West. When you spot land, you have arrived!

Beyond that, things get far trickier.

Rounding Cape Horn or Cape of Storms are both formidable challenges, even today. Without accurate East/West determination, you were left with far fewer heuristics and some mediocre workarounds:

As the Crow Flies – When lost or unsure of their position in coastal waters, ships would release a caged crow. The crow would fly straight towards the nearest land thus giving the vessel some sort of a navigational fix. 1

The race to map the world meant that enhancing location accuracy was as valuable then as it is now (maps, the new oil).

And so, a 1700s British government-sponsored technical arms race was established, matched only by the Data Stack sparring of 2021 in terms of strong narratives. The desired outcome - a better clock.

The marine chronometer (a much better clock) was conceived of and improved to such a degree that it enabled navigators to know where they were. 

The chronometer enabled better longitude determination. With an extreme emphasis on better (just a tiny snippet of the history of navigation, which may or may not be fascinating, I won’t impose).

The point, and why I find this a relevant thing to idle upon, is because this approximate location resolution process still exists. When you learn to navigate a modern sailboat, you are instructed to determine your location by taking 3 compass readings, resulting in a triangle, resolving to a fair chance that you are or at least were in that triangle.

One does this despite having a GPS, for obvious reasons to anyone who has ever relied on any highly available service - backup and validate.

And no, the numbers don’t match. Find peace therein.

The point is that within the context of “where am I?” there is always a certain uncertainty around how well all the data feeds are working. Quite literally in the rules of of the ocean, you need to continuously consult “all available means” to assure yourself that you know where you are. Granted the rule is for avoiding a collision, the point is that you may not presume to trust a single data point from a single system.

The link to the data

…is still coming, first another bit of history.

Sir Francis Chichester was another straightforward British navigator type with an entrepreneurial streak, having gone to New Zealand to set up a forestry startup with NZ Combinator. He later picked up an interest in flying, and went to the UK to buy a plane and fly it BACK to New Zealand over a few months - a new Tesla the modern equivalent I presume.

Through this long and dangerous trip (England to New Zealand in 1929 in the above single-seater float plane), he encountered a few setbacks in terms of knowing where he was.

Once in NZ, he decided to tackle crossing the Tasman sea, thought to be a bad idea at the time because navigation would be an issue. He needed to stop halfway across to refuel, and the tolerance for missing the refuelling stop at Lord Howe Island was near zero. If he wasn’t able to land there, he’d certainly be beyond rescue or recovery.

He was suddenly outrunning the metrics layer of a previous generation (neat), so he needed to develop his own:

The challenge of the Tasman remained and Chichester realised that he could reach Australia if he fitted [the plane] with floats to alight on the sea and refuel at Norfolk Island and Lord Howe Island.

The real problem now was to find these tiny spots in the sea. The only method of position-fixing available was to take sunshots with a sextant - not easy when you’re alone in the cramped vibrating cockpit - then laboriously work out the fix with pencil and paper.

He decided to use the principle of ‘off-course navigation’ i.e. deliberately aiming for a point to one side of the island. When this point was reached there would be no doubt which way to make a 90º turn for the final leg to the island. To reduce potential errors in calculation, Chichester worked out a series of examples based on his estimate of the sun’s position at the time of his expected arrival at critical points on his course.

The primary issue was around the speed at which he moved, but through a clever application of logic and tolerance for approximations, he managed (quite literally barely based on the reading of his book) to make it across.

He then went on to become the Chief Data Officer for the Royal Airforce flying school (training WW2 pilots), a prolific seller of maps, as well as a solo sailor (which is where he became famous).

The point here is that he was moving too fast for existing methods. The dynamic was changing too quickly for existing technologies. SQL Server SSIS was no longer sufficient. It may have been sufficient if you were doing milk runs in a steamship from Bristol to Londonderry (or a mid-tier bank with a CIO intent on minimal risk), but certainly not if you were trying to set a navigation record across the Tasman sea.

The Data Navigator

This is the link: Data + Navigation

Today, the problem of longitude has been solved to centimetre accuracy through satellite navigation2. However, the reason for the strong nautical-themed trivia, is that they demonstrate an excellent ability to manage and thrive with uncertainty.

Both Chichester and Cook knew the limitations of the BI tools of their time. They supplemented these with experience and intelligence to surmount what were pretty tough odds, to get to wherever they felt needed arriving.

For clarity, most navigators are indeed supporting roles to the Captain. The Navigator would use their combined technical skills, intuition and experience (collectively, wisdom) to give the Captain guidance on where and when. The Captain would then integrate all insights into their resource planning (a startup on the sea).

What is clear is that the Navigator didn’t rely on navigation aids as a crutch, but rather as means to better outcomes through a clear understanding of the weaknesses and an attempt at improving them.

So what is all of this leading to?

First, the Data Navigator is a useful analogy for startups and their data abuse.

When exploring, there is a benefit to be had from pushing into the unknown, and new methods are often required.

Second, bad data causes problems. Often it’s not the data, but rather the navigator.

Third, the navigator provides context which can be used as part of a higher purpose: alignment, motivation and maybe even consensus.

Three thoughts, three sections. Back to the sea, from whence we came.

1 - Exploration as a startup analogy

Navigation is well applied to the data team when the team is supporting the exploration of new worlds.

What the navigator does is supplement and confirm a mental model that someone has built about how the world works.

The map is not the territory

The idea maze maps nicely from historical exploration to startups - barely anything is known and there is a commercial framework based on discovery leading to reward. A startup is entirely lost, by definition, in the idea maze:

A good founder is capable of anticipating which turns lead to treasure and which lead to certain death. A bad founder is just running to the entrance of (say) the “movies/music/filesharing/P2P” maze …  without any sense for the history of the industry, the players in the maze, the casualties of the past, and the technologies that are likely to move walls and change assumptions.

The navigator has a theory about where they are and are going. This information complements the plan.

The startup has a plan of action and looks for confirmation or invalidation.

A navigator needs to understand their science in addition to the history and futures of their space. This becomes acutely more important when there is some form of race, as was typically the case with all historic navigation.

Innovation mixed in with luck and intuition

A great supplement to this is that the navigator doesn’t rely entirely on data. They use it to complement their intuition.

In a rapidly developing environment, the intuition delivered by a navigator gives an indication of which inputs can be trusted over others, what to prioritise and how to simplify the decision space.3

I think the Data Navigator concept echoes the purpose of data within a risky environment, where there is upside to getting it right.

2 - When it goes wrong

There are many documented situations where among other failings, huge ships just run aground.

He left her on autopilot, but strong currents overnight pushed the ship to the north and east and the chief officer altered her course towards the north. When Captain Rugiati awoke he saw that the Scilly Isles were unexpectedly off his port, not starboard bow4

The issue arises when the Captain receives information from the navigator that undermines the model. That number doesn’t look right. That island doesn’t look right.

Twitter bots, Substack daily views. These issues are easily overlooked until they blow up.

It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.

— ?

Dealing with these is the actual skill of the navigator. Being sure of what they know, and balancing this with what they don’t.

Navigators provide the best supplement to decision making, trading off speed against accuracy, and relying on intuition, and overcoming ingrained preconceptions and bias.

Speed not haste

Accuracy trades off against speed almost directly.

Crucially, this needs to be seen in the appropriate context. Some data teams function to ensure accuracy. In that context, the Navigator mindset is likely misapplied, and it’s probably better to rely on an auditor/historian mindset5.

Startups require speed, due to competition. Accounting requires accuracy, due to compliance.

Speed and hurry should not be seen as excuses for bad data. As with navigation, so with data, there are pretty firm fundamentals that are easy to achieve, can be effectively relied upon and supplement measurement and improvement of the basics.6

"To err is human, but to persist [in error] is diabolical."

— Latinum Sayinum

As a navigation route starts to mature and grow in traffic, then the balance favours accuracy, to drive efficiency. Similar dynamic when a startup becomes an established scale-up.

Hierarchy

It should be noted that the Nautical Navigator operates in a strictly hierarchical environment. There are no committees at sea, the Captain has the final word, typically for good reason, though the shipping disaster earlier indicates how this becomes an issue, as described in Black Box Thinking:

When we are confronted with evidence that challenges our deeply held beliefs we are more likely to reframe the evidence than we are to alter our beliefs. We simply invent new reasons, new justifications, new explanations. Sometimes we ignore the evidence altogether.

Many maritime disasters, when deconstructed, relate to power dynamic issues, indicating mental inflexibility around what was true. Running your ship into a reef or ignoring a warning about a storm are great examples of ignoring the data.

This is because new insights are disruptive and hard to integrate. People are happier with consensus than they are with uncertainty, even when the uncertainty might save them.

Organisations think they want more insights and innovation.

They are deluding themselves.

Organisations suppress insights for reasons that are locked inside their corporate DNA: organisations prize predictability and they recoil from errors.

— Gary Klein

Granted that the above line was in the context of innovation, not avoiding disaster, but the same “don’t rock the boat” mentality underlies both aversions to innovation and failing to avert course.

My anecdote - If you are on a UK train and someone is in your seat, and has the same seat booked, then at least one of you is on the wrong train - an uncomfortable insight.

The hardest thing for the navigator to overcome is an unwanted insight.

3 - Purpose

To bring things closer to the point then. A data team ultimately brings a shared context to an organisation. We are here, and aiming there.

As we know, the shared context is abstract, flexible, and transient. All the data does is add a foundation to the abstraction of reality we use to support and enhance our shared context. The data tends to be the one part that is consistent, reliable and infallible.

This has an underlying seemingly heretical belief: a single source of truth is an abstraction.

Single source of truth is a fairytale, data teams help reconcile this untruth.

What it looks like is shared context, with firmly drawn lines, in pen, on paper, that indicate that the boundary of the territory is exactly HERE.

Trails

And so what the data team does is simplify, create some abstractions, and identify options, and trails:

Complete freedom is not what a trail offers. Quite the opposite; a trail is a tactful reduction of options.

― Robert Moor, On Trails: An Exploration

This is to simplify and create shorthands, narratives, and reasonable decisions.

However, the Data Navigator needs to keep this simplification in mind, always apply their experience and skill to avoid disaster, and ensure the drift from reality to map isn’t too severe:

The map maker, the surveyor, the compass reader, the ships crew, and everyone that had some part in the determination of the final pen to paper knows that there are many sources of uncertainty, each possibly extending the truth in their own limited capacity that may indeed lead to the pen on paper determination being slightly but disastrously off.

We had 424,000 daily active users yesterday.” The pessimetricist thinks — hopefully, he does not say this — “Actually, you had in excess of 424,000 HTTP requests from devices associated at least temporarily with unique user accounts registered in your internal systems over a 24-hour time period that survived a number of arbitrary assumptions in your data processing systems that passed muster six months ago but which haven’t been re-evaluated meaningfully since.

— Stephen Bailey

But pen to paper it is, and from that moment onwards, and until a better map is created, that is the single source of truth, the true story.7

Certainty Matters

If the way we are collecting data, storing data, transforming data, distributing data, consuming data and then sharing data each have between 0.1% and 1% chance of error, then that error gets accumulated, and in some cases amplified, apply a narrative and suddenly the truth could be anything.

Is the data correct? What is correct? What is?

This doesn’t land well with people on the receiving end of data systems and old maps. Accountants doing financial reconciliation on data warehouses, and someone who has smashed their fibreglass boat into an uncharted granite rock8.

When a data team gives an indication of uncertainty - this causes thoughts along the lines of YOU CANNOT BE SAYING WHAT YOU ARE SAYING.

The map, still, is not the territory. It is part of building consensus and alignment.

In the same boat

Context isn’t enough, one needs consensus:

Data professionals can build consensus as the company becomes more diverse. Data systems can establish methods for understanding the world even as it becomes more complex. [My emphasis]

— Stephen Bailey again! Perennial Truth Architectures

What that means, is that using context (meaning), the navigator and the team need to build from that shared context towards consensus (opinion or position reached by a group as a whole).

Without consensus, we get stuck in the wrong quadrant of a 2x2 matrix, where autonomy and indirection lead to the night watch sailing the boat in one direction and then the day watch on handover reversing course and backtracking. Consensus is king.9

Last Words

To wrap things up, the story here is as follows:

1. Help your team/company by navigating. Reconciling uncertainty is your special responsibility.

2. Hierarchy and inertia are cultural issues that flummox good intentions and amplify bad data. A purely technical orientation will only get the message so far, a voice and an opinion are necessary to succeed.

3. Build a shared context, and use it to reach consensus.

Navigation [Data] is easy.

If it wasn't, they wouldn't be able to teach it to Sailors [Business People].

— James Lawrence

This is a story about how you need patience to let the barbs catch.

(What is a foil)

The first time I ever saw a foil in the flesh was in 2014, working on a super yacht in Phuket. An older couple were taking turns towing each other behind a small boat. At the time I thought I wish they’d give me a go, it looked like they were really hacking at it. With the confidence of someone who was effectively a professional boat driver and cleaner, with a lifetime of dragging people behind boats on nearly anything, this looked like easy money to me. They never came back and I was too busy chasing barrels in Bali for it to cross my mind again, and so the barbed hook missed me for the first time.

The first time I tried a foil was in 2017. Clifford of UniFoil had been in touch with my dad about something to do with surfskis, and it surfaced that he was building a foil. We got in touch and offered our boat to do some towing in the river in exchange for a go. We have a few cracks at it, I nearly swallowed the foil wing whole in a jack-knife fall, and Gumby who had a few hours on it over us even managed to surf the tiny wake. My brother linked up with them the next day and had a go at surf foiling. I was already back in Cape Town as I had a new job, was recently married, was building RC gliders, and so slipped the barbs once again. I was definitely tempted as Clifford/Unifoil was even offering a local orders discount. I asked him if he had any used setups going cheap, but never heard back. Back to Cape Town I went, without space for another consuming hobby in my life1.

Fast forward to 2021, having moved to London (not conducive to anything other than indoor bouldering) and then Oxford (even more landlocked!), I found myself in a riverside apartment for the summer. A friend from Zurich instructed me to get an Axis 1150. Foiling is happening, he said. These things were not cheap, in fact they are blindingly expensive, but somehow fate handed me the only used 1150 I’ve seen listed. Some hyper aggressive scrounging and I managed to put together a reasonable, if metallurgically tired setup. I was going to learn to dockstart.

---

I have a theory for how to spend your time: do what is going. (what is going? Do that). This means that based on your current and near immediate future, choose your current hobbies and interests based on what is optimal for that present and future. Do what is going. Wherever you are in the world, find the thing that catches your interest, gets you excited but most importantly is appropriately selected for your life/lifestyle. When in London, bouldering & cycling are appropriate. When in Cape Town in summer, don’t hate the 3 month long 40kn gale, ditch the surfboard and get some kiting in.

The advice is to be water, dear friend. Go with the flow rather than dogmatically hounding your calendar for your next trip.

Advice easily given, and yet poorly applied. I no longer kite and so spend many months in Cape Town eeking out icy surfs while the wind nukes. In London I spent hours getting to Southampton to crew on racing yachts. You can’t reason with your own curiosity really, but what you can do is use this theory to justify something new. And so applying my theory to myself, living 100m from the river Thames, with a push from a friend, the challenge of learning to pump a foil off the dock felt like it was right. 

---

Having not seen a foil since 2017, and with a grand total of well below 5 min of foil time, I was cautiously optimistic that I would just nail it. This being a semi-reliable hunch, as the more things you crack, the easier the incremental new things become.

Not so with the dockstart. I stand by the 30 attempts per day for 3 days2 as an approximate minimum effort to get over the first hump (not falling immediately). Your brain needs time, and especially sleep to synthesise the new experiences, and eventually you just hit a wall each day. Fascinating.

Save yourself the trouble and spend an hour behind a boat if you have never foiled before, or just go for it - learning and getting spitting mad while doing it is part of the fun. 

Cracking dock starting is possibly worthwhile. It is ridiculously challenging initially, but eventually your brain just rewires itself, and you can cruise, work on efficiency, strength, technique in order to get beyond the breathless bursts and up to 1 minute flights and beyond. The main value ultimately, as jumping off the dock is a means to an end, is that it gives you the very useful sensitivity to low speed near-stall riding, and very quick feedback to the feeling of different setups. The only casualty was a few gashes from The Knives as my brother likes to call them, and a nearly toasted 4/3mm wetsuit that took the brunt of the 1000’s of times elegantly climbing (flopping seal) up the concrete jetty.

What the dock doesn’t teach you is to deal with the dynamic and turbulent sea, least of all the incredible surges of energy in a wave.

---

Cornwall was always the final destination, and with the foil hooks starting to bite, we headed to that rugged North C. coastline. This being Autumn, the surf started to turn on. The foil was parked. The 6’1 was freshly waxed with an Insta worthy base-coat/top-coat textured landscape, and the mind focussed itself on chasing big walls of raw Atlantic energy dragging themselves over reefs along the coast.

In the back of my mind the foil was still there, but only having the massive 1150cm span wing and a kiteboard, surf foiling eluded my invalid attempts, stuck as I was between the urge to buy a surf foil setup and the desire for a new mid-length board.

Front of mind, the mid-length won.

Foil gear sits near the “how much!?” range for most, and the inflexible nature of each bit, and incompatibility between brands, is really all just stinging nettle in a surfer’s wallet. Up to £1000 for a board and nearly the same again for the foil bits I needed just wasn’t happening, especially when a mid-length in SA costs less than a foil front-wing alone (a used one!). 

Back to South Africa for December, with the pump foil still in tow, but not really getting a look in over the mid-length and a few runs of good surf, and a freakish meaty swell to light up some beaches. The kiteboard that I brought with got wet once in the sea, and a few dock sessions, but it really wasn’t working on that board, and I was loath to part with any more money.

My hometown of St Francis Bay is pretty well renowned for fun easy surf, and is high on the list for local foilers. This has led to a few grumbles from the locals at the prime longboard surf spot, frequented by all craft, all ages, and probably has become too festive with foils thrown into the mix.

The whole town is oriented to water spots, and so leaving the foil aside, I surfed my dad’s new noserider, the first time I’ve surfed a longboard with a flatter tail rocker, and it was quite perfect. Equally good was the malibu surf rescue board. I bumped into the local foil crew and chatted shop about buying one of their progression boards, and considered chatting to the local surf-pro who looked to be foil hooked, but the spots were all rammed with tourists and the surf was good enough.

Clearly the foil's hooks were slipping.

The foil does work in mysterious ways, and someone my dad knew (or knew of) had bought the Flite efoil franchise for SA, and had got chatting when he paddled past. “Don’t ask, don’t get” applies here. Turns out he was keen for people to try it out, and so the 3 brothers paddled over to the slipway that next morning for a go. Efoils are like jet skis, best to have a buddy who owns one.

The novelty was epic, but the appeal wears thin, rather it feels functional, like a jetski. And so another barb slipped, and I headed back to winter in Cornwall, ready to challenge some winter walls.

---

It is no secret that the North Atlantic slows down in the summer. If you’ve grown up surfing the Southern Oceans, you’d be excused for not believing quite how flat it gets. The winter months see the full brunt of the gulf-stream born storms hammering straight into the coast, month on month of swell, and the wind backing just frequently enough to give the attentive surfer some dreamy cold water moments while the tide is just so (whipping up and down its 8 metre range). 

Suddenly it stops. 

Late spring sees a big high pressure system parking itself in the North Atlantic and deflects and neutralises any semblance of a storm, only the odd hurricane pulse can traverse the gap. This leaves the western Europe coast, at least from a surf perspective, in a state of lake-like tranquillity. From ravaged sea cliffs to inviting coastal swims. Polarised is the word.

And so with this in store, I felt the stars starting to align, and a surf foil would start to sell itself in my mimetic subconscious. A few things needed to line up for this to work, and allow for the metamorphosis to take place.

Primarily, it was viability. Are the circumstances correct and ready? Alluding to this theory earlier, the circumstances had up until this point not been ready.

Now, close to the beach, a few GBP saved for a used board, nearly flat ocean. Perfect.

---

Next on the list is more subtle. The Scene.

The best stoke in life is shared stoke, and there is nothing quite like the shared froth of a bunch of people facing into the unknown possibilities of a new thing to learn. This is amplified when the thing itself is experiencing an inflection point in its growth or innovation.

One of my formative experiences was the sudden growth of downhill skateboarding in 2009, of which I and a few friends were reasonably influential (before it was a thing). This surge was triggered by many factors, technological, economical, but what lit the spark was a few video clips showing a new way of doing something otherwise quite familiar3.

Something I reflect on often about that skateboarding time is that I spent the year prior to getting involved in skateboarding working on a yacht that was doing a refit in Mallorca. The year was spent mostly discovering life as a newly minted yachty, and while I dragged along a longboard (and making a replacement when it snapped, in the lazarette of certainly one of the most beautiful classic sailing yachts in the world), I never bothered to notice that Mallorca has some of the best skateboarding in Europe, and home to the only 360deg corner I know of. Skip forward a few years and I was straight back there, amazed at how perfect the hills were.

The point is, neither skateboarding nor I was in a position for that level of attention. I had all the time in the world, but I was on an adventure already, and didn’t necessarily need nor was I looking for additional ways to hurt myself in the pursuit of adrenaline.

I often wonder what my skateboarding career would have looked like if I had pursued skateboarding with the focus I did pursue it, but starting exactly 18 months earlier, with literally every race I would come to dream of within a few hours travel.

In the end, with luck and enthusiasm to thank, we led the charge of South Africans in the downhill racing scene, and travelled to a few continents, but mostly the Alps, shooting video parts from the nascent growth, right up until 2015 when it hit a relative plateau, and my interests moved on, my ligaments grateful. Downhill skateboarding has many ebbs and flows, like skateboarding itself, but this was the first time it hit mainstream for a brief blip.

Regardless of where foiling ends, the scene is what sustains and drives the hook deeper.

Foiling is at this point. Once (still?) a kooky fringe activity, the estranged child of SUP. Foiling is slowly developing a coherent style, and eventually it may even establish a place in the line-up.

---

Presently, I am finding my glides along the North Cornwall coast, finding the best chip’n’rip opportunities on a low tide, up the beach away from anyone. Foiling is at a point where it is still rare enough and early enough to mean that everyone at my level (linked my first few waves as of writing this) is a likely candidate for shared sessions, shared tips and shared stoke. Myself and another Matt are now partners in bump hunting and gear tuning . A sandbar that fades into a hole is optimal, giving us an easy entry and then space to turn the massive dock start oriented foils out to find another wave.

The gear I have is enviable to someone who has no gear, while frustrating to me in it’s lack of relative performance, and causing much time lost, foil-brained to the max, in pursuit of what might speed things along, as seems to be the case with all the foilers along the North Cornwall coast, a tight knit crew with a friendly attitude, known as the KernowFoilCrew.co.uk

The best solution to the foil-froth-brain feeling is more time in the water, trying smaller waves, try try try other people’s gear, but ultimately, I have found no shortcuts.

What is clear is that the humble foil might finally be having a moment. In the greater scheme of things, people were foiling as a hobby all the way back to the 90’s - with pedal assisted foiling canoes and all manner4. Readers may wonder (if not already hooked), if now is their moment. If the water is ready. To that I would say, let yourself be drawn in. If you find yourself hooked, let the hook settle. 

The rest of us that are in various states of the flow should maintain perspective. Nothing lasts forever, this too shall pass, and life will go on. That said, embracing the present is what it is about. What a special thing to be a part of. From my experience with downhill skateboarding, a combination of things lead to a plateau - gear performance plateau, saturated market, public pushback, governing body collapse, trends changing, bodies aching. 

With foiling, the key risks are likely negative perceptions from surfing, and a crowded foil line-up won’t get anyone stoked - which we are likely closer to critical capacity than we realise.

A note to the other things that have their hooks but have yet to pan out, remaining on the distant horizon of possibility. To me, this is sailing, far away. These things sit waiting patiently, possibly forever, but will also sink a few hooks if the moment is right.

Foiling is here. The line-ups are probably not ready for a critical mass of new novices, and etiquette, rules, lawsuits and probably worse are likely still in store, but this post serves as a tribute to all the curious tinkerers and surfers who have bent aluminium and carbon to the current point, and especially those who spread the stoke even further.

As I write this, having completed my first 3 wave link (entirely marginal link, and after far too many attempts, and a serious limp from getting ridden over by my foil in the shorebreak), I wondered who this is for.

If you’re hooked, commiserations and joy to you, your attention span and your wallet.

If not, then this is maybe for you. Nearly everyone who got hooked was somewhat sceptical, maybe even had a go or bought a foil only for it to fall to the side. You can’t force it (you could I suppose), but when the time is right, let yourself take the bait, and run with it a bit.

1

Another piece of irony that foils are entirely aerodynamics nerd heaven, and I lost interest in RC planes pretty quickly because of the lack of physical challenge to complement the very interesting wing design elements - washout, foil section, dihedral, reynolds…

2

If you’ve never seen a foil before, don’t beat yourself up, I estimate I must have had around 1000 attempts before feeling any semblance of control.

3

Early GoPro on a stick pioneer, to my credit. Netsky even met me with some tickets to his SA concert.

4

Laird in 2003, a race in 2000

Edit: 2022 was the flattest summer ever

Trigger warning: this may trip your thought leadership nerve, but I’m mostly riffing, and I think it is additive. This post was easy to start, a pain to finish and lots of fun in the middle.

This is a narrative for the near future of Data Engineering in startups, and I think makes some interesting points. I do think the post has avenues for expansion, especially counter-arguments in the context of enterprise tech. 

Hello to new subscribers, and a shoutout to anyone using the RSS feed.

Thanks to the LocallyOptimistic.com community for some lively discussions on an early draft of this post.

Intro

The core premise of this post is:

Most businesses' data engineering needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools.

For the exceeding majority of businesses, this means they can and should focus on building capacity for business logic, analysis and predictions instead of data engineering.

The minority of businesses that need streaming services / low latency batch data, will further push the boundary, using specialist Data Engineers.

The implications are that while Data Engineering is growing rapidly, so too are the forces that will undermine the need for Data Engineers, and the current under-supply of competent engineers will lead to an over-supply of junior engineers (this should ring a bell to the Web-dev then Full-stack then Data Science boot camps).

Let us break down the premise further, as that is a massive generalisation, and to the reader of this niche corner of opinion, may seem counterintuitive, inflammatory, and frankly, stupid. 

6 points, expanded into 6 sections. Let's go.

1. Majority

Keep in mind the context for the majority of businesses - technology is often an expensive misdirect when implemented badly. This is exceedingly true when the technology is not directly aligned with their competitive advantage. Majority here means all businesses investing in technology, not just the typecast "blitz scale tech business". All businesses should take advantage of data tooling, or they will in effect be flying blind relative to their peers (or get an advantage over their peers if they get it right).

2. Data engineering

Plumbing of the data - ETL, data warehouse, streaming, batch, orchestration, infra etc. Niche skills that are hard to hire for. 

Distinct from Software Engineering. Mostly not backend developers, more commonly generalists, occasionally highly skilled specialists. 

Data Engineering is quite contentiously named as is made clear later.

3. Business logic

This is covered elsewhere - but in the most abstract terms - businesses should hope that their engineers’ primary focus is on improving the ability to represent the businesses various states, and enhance the ability to interact and modify these states, and even predict outcomes to modifications.

Businesses should strive not to have people worrying about managing infrastructure, plumbing, ops etc over and above what is strictly necessary. Playing on the margin of this point is what the CTO does. 

4. Managed services

Think about Sysadmins of the mid-2000s, arcane knowledge that is now redundant in almost every business, due to AWS, then Heroku, now Vercel, Supabase etc flying up the stack. (Or hadoop specialists. Big Data DBA anyone?).

Same with Data Engineering. Tech abstraction as a service. Managed Services are arriving fast with the likes of Snowflake, Fivetran and the commodifying follow-ons. They’re aggressively chasing down the almighty dollar, undercutting margins, offering better cost structures, as well as a flurry of bundling mergers and consolidation.

5. The minority

Many businesses will still have an exceedingly strong need to increase their advantage through data engineering. Take High-Frequency Trading as an example. These businesses will progress the field, and the best data specialists will be needed in those spaces.

6. Implications

This one is clear, don’t get caught on the wrong side of any sea change. I would (do) argue that the ETL engineer skill-set is mostly going to be marginalised until :skull:.

In much the same way that the market demand for boot-camp Data Scientists is low, due in part to oversupply, better tooling and additionally a reorientation around the expectations of a Data Scientist, so too do I propose that Data Engineering demand dynamics will change. 

I'd like to hope that the rate of ETL code being written is in decline because most can rely on managed services or open-source ELT extractors. 

This point gets some pushback, discussed further in part 6.

But more generally, here is something about changes in the tide:

When the tide turns, there is a definite moment when the tide has indeed turned, but that change in direction becomes apparent to different boats at different times. This depends on context, location, keel depth and distance from both the equator and the moon (not to mention the sun). The gravitational pull has changed, but the water doesn’t start moving everywhere at the same time. 

This blog also, as it became clear through writing it, and quoting sources, agrees with a certain viewpoint on specialisation, the link shall possibly become obvious.

So, that is the intro, I’m going to stick with those 6 sections so that coherence abounds, and explain / expand the points in finer detail. 

1. Majority 

Nearly every company needs a data person. Any company that has ambitions to beat market returns on their investor’s capital and doesn't have someone in a broadly data-dedicated data role will certainly struggle to compete.

10-15 years ago an easy indication that a company wasn't keeping up was having no IT person, the modern equivalent is having no Data Person. 

But the premise is that that person no longer needs to be a Data Engineer.

Reference the sysadmin/dba type roles, which for 99.99% of businesses does not exist, because cloud providers hire those people and abstract their role into a service.

The thrust is that Data Engineering could go the way of the dba. Niche, specialised.

Who/What is the data person then?

Data engineering in the ETL/ELT sense has historically been complex, difficult, emergent, at times chaotic, and required niche software engineering skills.

Now, Extract and Load for most businesses using generic SaaS tools, is solved. Using the standard set of CRM, HR, Finance, and Ops tools, 80% of your ELT work is done for you at a standard, predictable price. 

Commoditised EL SaaS is ubiquitous, with the second wave (of EL providers) offering better services at more favourable terms than Fivetran, with multiple variants and mutations.

T for transform, with general best practices courtesy of dbt, is where the bulk of the  analytics work lies. Critically this is where the data person should start. 

They should be, to some degree, what is known as an Analytics Engineer, but possibly more usefully, not a specialist Data Engineer (nor a Data Scientist for that matter, but that bridge has been crossed).

They should be a purple data generalist:

The data world needs more purple people — generalists who can navigate both the business context and the modern data stack. Let's put aside skillset dichotomies, and learn to feel comfortable in the space between.

If you do need a Data Engineer, probably for some or other niche API that isn’t supported by your EL tool, then this is great work to outsource! (My day-to-day work: Supporting companies on their fringe data engineering needs when their internal team wants some extra capacity or capability).

But the first full time data hire needs to be obsessed with business impact.

2. Data Engineering

A tale of two types of data Engineers: Again with the generalisations! In my view and from what I've seen in the job market, there are two types of data engineers at the moment:

(1) Data Engineers: Software engineers, Data

Described as: Software engineering specialists, with data as the core specialisation, who can focus on the niche areas of data engineering and can work with complex real-time data systems.

Needed When: Only required in tech businesses, and only when software engineers cannot assist. This is not needed for 99% of businesses and these candidates know what they want to work on and have the agency to decide.

Characteristics:

• Tools-oriented

• Computer scientists / Very good software engineers

• Driven by curiosity

• Driven towards perfection of the craft

• Want the solutions to be elegant, optimised

• A specialised role for a specialised business problem

Currently and into the future hired to do the following (quote from a slack group from someone who may or may not want the shoutout):

When building out some data-focused applications, like, say, a streaming data enrichment layer that serves up some curated data real-time to other micro-services, we need software engineers and data engineers. Occasionally you’ll find unicorns who can do it all (we have a few of them), but the vast majority of software engineers aren’t experienced enough with data to also be able to solve complex, big-data, non-SQL problems as well as someone more specialised could.

(2) Data Engineers: Solutions oriented engineers, Data

Described as: Business optimisers. Data engineers that engineer data because it is the biggest blocker in the optimisation of a bigger picture issue, namely analytics as it relates to business improvement efforts. I love this post from erikbern.com:

You work with the recruiting team to define a profile for a generalist data role, that emphasizes core software skills, but with a generalist attitude and a deep empathy for business needs. For now, you remove all the mentions of artificial intelligence and machine learning from the job posting.

Needed when: Data engineering data extraction and centralisation is identified as the key issue in a long line of issues. The primary bottleneck in the optimisation process.

Characteristics:

• Goals oriented

• Background in an adjacent engineering field

• Driven by optimisation, the ultimate goal

• Utilitarian problem solvers, relied upon to get the job done

• Functionally broader skill set, maybe even new to the domain, and not (yet) experts in technology

Historically hired to do the following:

If you were on a “traditional data team” pre 2012, your first data hire was probably a data engineer. You needed this person to build your infrastructure: extract data from the Postgres database and SaaS tools that ran your business, transform that data, and then load it into your data warehouse.

Currently hired to:

Build data warehouse, pipelines, dimensional modelling, deploy analytics tools, string it together, but critically, to drive change in a business.

In short - Type 2 wants the solution to be cheaper, easier, faster, best fit, 80/20, is less intrinsically interested in the how and more interested in the impact on outcomes. 

In my opinion, this is basically now Analytics Engineers, and if you disagree with my take on this concept, speak to an Analytics Engineer who had the title Data Engineer, and ask them if they can relate. Similar experience to those Data Scientists who preferred Analytics Engineering.

Another way to think about these distinctions is (erikbern.com again):

I often think of people as (and this is an unfair crude generalization etc) roughly on a spectrum between tools-oriented and goal-oriented.

Memed as:

3. Business Logic

Engineers as optimisation specialists

My background is in industrial engineering, which is broadly a stats'y engineering field incubated in the optimisation of systems (typically factories).

Layouts, flows, bottlenecks, JIT, supply chain etc. Mostly a solved field in many regards (shout out to the bullwhip effect 🚛 🚚 🚛).

The broad optimisation process for most of those businesses looks something like

1. Collection of SaaS and ERP-like systems to track and account for things

2. Data engineering to extract the various states

3. Analytics on the states

4. Decisions to change the states

5. Track decisions in ERP (i.e. repeat)

When I left engineering school, ERP implementation was where the demand was, and large chunks of engineers ended up implementing/consulting/suggesting various guises of a(n) ERP / CRM / database / app / spreadsheet / chalkboard.

However, it became clear to me (with hindsight) that this was quickly becoming a commodity technology and skillset (i.e. outsource to contractors), and that Data Engineering was the real skill bottleneck.

Businesses were amassing large data sets but struggling to access them, let alone analyse them, and so having lucked into a DE role, I made this transition.

Optimising down the optimisation list

Data Engineering is no longer the bottleneck! This is a huge relief, because Data Engineering is not optimising, rather just a necessary lift and shift. It is purely an operational burden brought about by decisions made with siloed data as the tradeoff. 

Now surely 3. Analytics on the states is the biggest hurdle and opportunity. 

Analytics is currently a headache, which requires significant investment, and where I suggest the investment is made. There is much more value in time spent on the building of “Business Logic”. In this case, Analytics. 

Analytics extends far beyond data modelling and analysis, encompassing business processes, people processes, management and communication. 

Analytics also pushes back into software engineering, system designing and overall value chain analysis. 

4. Managed services

The Data engineering "type (2)" makeover

My day to day is where I form this opinion. I have done much less data engineering as it relates to Data Warehouse fine tuning, and ELT troubleshooting since the tools became so much easier. I do a lot more analytics and a lot more modelling. The problem has moved, onwards, up-system. The old bottleneck has largely been removed and solved. Optimised.

To quantify this stance, consider why  there is a literal tsunami of new spins on data products: metric-stores, reverse ETL, metadata, discovery, quality, etc etc. Great data from Benn Stancil:

In 2017, Y Combinator—an incubator of both startups and the Silicon Valley zeitgeist—funded 15 analytics, data engineering, and AI and ML companies. In 2021, they funded 100 (my emphasis)

These are viable partly because the EL bottleneck was eased, the storage got cheaper and dbt made the whole thing more manageable.

Suddenly the problem wasn't getting the data, it was using the data.

Typically the domain of the elite. The reason Airbnb, Linkedin etc have needed a data catalog for near decades is because they had the engineering clout to make it necessary.

The sudden simplification of this process has meant that the next, hitherto unknown bottleneck gets suddenly bashed into, and there is immense value to be gained by unlocking it.

Build it, will they come?

If offered, many businesses will jump at a SaaS subscription, rather than spending that money on hiring/expanding an engineering team.

The term engineering is derived from the Latin ingenium, meaning "cleverness" and ingeniare, meaning "to contrive, devise" wiki

When the data is easy to centralise, combine and analyse, engineers won't be needed to devise and contrive data combining solutions.

They can go and contrive and devise something else, that is complex, and that gives the company a competitive advantage.

Eventually, analytics engineering could face the same turn of the tide. When the tooling gets so good that the team is composed entirely of analysts and product people, and no contriving engineers.

In the same way that structural engineers are only required when building on quicksand, data engineers are only required when building upon a dataswamp. As the tooling gets better, so do the foundations stabilise.

On the margin

The companies I advise and work with often have much less need for Data Engineering at the outset.

However to clarify one point - when they do need Data Engineering, it is a requirement for specialisation. There is indeed more Data Engineering to be done, but this is increasingly specialised (this is a semi-deliberate contradiction to this entire post that I am OK with).

The companies need help with the edge-case, marginally viable solution, where something emerges, crucial to them, that falls through the cracks of the 80/20 SaaS solutions. The point is that these needs come later down the line. Not at the outset of a data project, but later, once the bulk of the crucial, impactful elements are working and generalist data practitioners have exhausted their options.

A caveat: the assemblage of the appropriate tools in the appropriate order to match business needs and maturity is a tricky problem indeed. Probably something that would benefit from the skills of a Data Engineer. More on that in section 6.

5. Minority

Data ENGINEERING isn't going anywhere.

I recently discussed this with someone from a quant hedge fund, and while they had a computer science background, they were "data" + "engineering" to a profound degree. They needed a real-time (real time real-time) data feed from all of the brokers, with extensive transformation across all of them. Multiple decision systems integrated with predictive models, and then reliably send orders back into that system, in near real-time.

This system literally was the business. Complex, differentiating. Building this was one of maybe two things that the company needed to execute to beat the competition.

Data engineering in certain contexts is necessary, but likely to be a specialisation increasingly of interest to the minority.

The above point alone isn't that contentious.

What is contentious is the WHEN.

Has the tied turned, is it still rising. Who is seeing the signs and who is missing them. Who is seeing evidence where there is none.

6. Implications and Evidence from the field

Hello?

2 more minutes, less hand waving I promise.

Implications for Engineers

This entire post makes the same point as this specialisation bombshell:

What is the right level of specialization? For data teams and anyone else.

It seems fair that, if tools didn't require so much knowledge to use (I'm looking at you, Kubernetes), then on the margin, the need for specialisation would be less.

The extension of this point is that because the data engineering toolset got so much better, the specialisation required is now less. Snowflake and BigQuery users agree. 

The implication for engineers whose work is now easier is the following:

Either you move in the direction of the new business problem.

Or you move to a new business that still has the old problem. 

Or you specialise further until you find another domain to play in, and wait for the tide to turn again.

Erik's blog above makes another point, which made me realise this is a mostly “deeply inspired” notion, so much so that I've used the tools/goals oriented concept in an earlier section.

I often think of people as (and this is an unfair crude generalisation etc) roughly on a spectrum between tools-oriented and goal-oriented. Some people have their favourite tools, and that's what they like to use. They make their whole career about honing a craft with those skills. Other people are more entrepreneurial, and don't care about what tools they use: they care about the ultimate goal.

This topic was quite contentious on Twitter. People made some very stern remarks about specialisation when Erik posted it initially, and I guess I'm not surprised. People are very likely going to fight against any concept that undermines their career domain.

However, this contentiousness further highlights the opportunity:

Contrarian ideas, when right, are "the valuable thing" from the Taleb and Zero to One books:

“What important truth do very few people agree with you on?”

Should this point be right, it will be proven right by (another) tool that reduces the need for specialisation and sells for ${LOTS} because it enables achieving Goal X (data-driven-whatnot) without hiring a team of 100 ludicrously demanding human specialists with endless needs.

Arguments, of which there are a few, against this, include that the startup ELT paradigm is a minority and that data engineering work is firmly entrenched in the structures of larger businesses, especially enterprises. The refinement I think is worth making, is that while this may be true, the hope is that it will become less so. Like the shift to the cloud, I would hope that what we describe as ELT now leads to us finding a better way of doing things, whatever it may end up being, that is as transformational for Data Teams as cloud computing was for Software Teams. (Noteworthy that “hybrid-cloud” has proven so popular with enterprise)

And a pushback to this: enterprises aren’t most businesses. Most businesses don’t have a large tech team, most businesses didn’t exist a decade ago. However most Data Engineers are not employed by most businesses, hence this disconnect. Most Data Engineers would disagree with this premise, but the point is that most businesses won’t need a Data Engineer. 

Looking at history, this happened before, take a look at Data Science as a field, maybe due for a renaissance in the guise of ML. “Data Science” was a crutch for companies not knowing what to do with their data. 

Implications for Businesses

The message from the communities and my experience is clear - Data Engineering as it once was is generally less of a challenge - but building a coherent “data platform” remains a chore. 

What is possibly the most complex part of “Data”, and what they really need help with is, what I suppose quite fairly is called Data Platform Engineering:

• EL tool can start costing inordinate amounts relative to the value gained. 

• X tool sunsetting Y feature 

• Adding a new business tool with an unsupported API that needs a singer tap built. This work typically is open-sourced, so eventually, there will be fewer needing singer taps (pray)

• Airflow proving to be a headache. According to Slack, 90% of airflow users are using managed services, so less specialisation in airflow will be needed (pray pray)

As an example of this, I’ve recently consulted on the best way to ELT some data from a few API sources unsupported by Fivetran, as well as Stitch/Airbyte. The decision complexity is quite high:

1. Is an orchestration tool such as Airflow/Prefect needed yet, and if so, which one?

   • If Airflow, then the AWS instance, the Astronomer version, or self-host?

   • Do we try at the outset to use Kubernetes? Is Airflow stable yet? It still feels overcomplicated.

   • If Prefect, will they as a new entrant be more reliable or still have teething issues?

   • What level of CI/CD for the tools? 

   • Would they benefit from Terraform?

2. Meltano, Airbyte or Singer extractor/tap spec?

   • Meltano [1] seems to be making excellent progress, but requires some minor hosting effort, and also requires an orchestrator. 

   • Airbyte seems to (seems to) be making more of a commitment to quality.

   • Both are wrangling with the ways of incentivising community maintainers.

This is just one “component” of the team’s ELT, not even the full picture of their Data Platform, and it is a subtly complex and consuming decision for those familiar. 

A great way to frame this is quasi-architectural DataOps flavoured generalist Data Guru role of the Data Platform Engineer (DPE):

DPE are thinking about what data exists, who should have which access, how to make it available for usage by people and tools, how to make it redundant (disaster recovery), how to enable discovery (catalog), etc   

Or another spin:

DPE just means that you are the Tech Lead of the Analytics Engineering.

While I don’t necessarily care for the DPE term over DE, I do think DPE aptly captures the key work that many Data Engineers now do, combining and ensuring cooperation between competing tools to build a coherent consumable data platform. 

More than anything, the developer experience for most of the necessary Data Platforms tools is just garbage. Airflow is a nightmare, GCP really a frightening pain, and AWS is just so much worse. The correct abstractions over all of this is a huge opportunity and the thing that the DPE needs to keep an eye on.  

[1] Worth your time to have a look at the Meltano SDK if you need to build an API extractor. Great team, developer experience and ambition. If you are a Data Engineer (either type), these open source projects are possibly the best intersection of your skills, interests and market demand. I set up the most lightweight way to run a Meltano ELT on AWS, using Terraform, and could use a review!

Closing

In closing, I broadly see the below chart as usefully inflammatory and marginally useful.

As Data Science gave way to Data Engineering enthusiasm, I'll say that Data Engineering enthusiasm possibly will have to give way to Analytics, currently called Analytics Engineering.

Following this will be the traditional Data Analyst role, in whatever new guise, which will make some resurgence.

However, the core Data Engineering skill-set, technological awareness and systems thinking, will remain vitally important, but perhaps not in the historical and existing notion of a Data Engineer. 

Appendix

Questions to ponder, hit the comments if you have some thoughts:

1. Will data science re-emerge now that the data wrangling tooling is getting so much better? What will this do to the hierarchy of data science? Maybe ML Engineer is a better candidate.

2. Where does MLops sit, it largely has felt disconnected from “Modern Data Stack”?

3. The enterprise dynamic is entirely different. Enterprise companies will need ETL engineers until the heat death of the sun, and no I don’t want to hear about it.

4. Is training Data Engineers a lost cause, along with Training Data Scientists, Front-end devs?

5. Remember that 92% of startups disappear, but while we are stealing fun from tomorrow, we can satisfy ourselves knowing that someone will get it right, but for someone to be right someone else must be wrong.

Please consider subscribing for more on the subject of data systems thinking

What is group by 1

Who is Matt Arderne

The barrier to entry to working as a data analyst is reducing rapidly. If you are a combination of curious, technically astute, outcomes-oriented, driven, observant, people-oriented and a natural leader then the technology should not be a hindrance, as it is quite literally being made easier every single day.

Consider this as a primer, and reach out to me directly if you think you’d like to work in this space, especially if you are motivated and have any data experience.

Is it a good fit for me/you?

1. It is a great fit for people with good soft skills, strong intuition for business operations and a desire to make a difference in how a company operates. Are you interested in finding out the facts about what happened in the past and suggesting changes as to how they should operate in the future? To tell the CEO this? To back the decision and measure the changes? To possibly have been wrong? Good things to like the sound of.

2. There is Very Strong Demand in the startup space for these kinds of roles, so if you can convince the right person that you have what it takes then it is entirely possible to transition into this career, without any formal qualification. Typically good transitions are from engineering and technical operations in fast moving businesses.

3. With that goes the obvious implication that the current high demand may not be sustainable in the long term. 

4. The term Data Scientist has largely been merged into the analytics roles, as pure data science is very niche, and often misapplied. Typically companies need more data analytics than they do data science unless their core product is somehow related to data science, and even if they say it is, it often in reality very much is not.

5. I prefer working in or as a provider to startups and scaleups, as this environment is more dynamic. Larger organisations and enterprises have analytics functions, but they often have more functionally or departmentally specialised roles, which means less exposure to the business as a whole. As you become more senior, larger companies will provide a great growth opportunity. 

Data Analyst & Analytics Engineer stuff:

Necessary tech skills.

1. The Data Analyst / Data engineer concepts have been pushed together into a single role, Analytics Engineer, which broadly makes sense, and can be considered a role related to the Data Scientist. I explained this to some extent visually in this tweet. Further, this describes in a bit more detail what we do and don’t know about analytics engineering.

2. These roles are generally distinct from or work alongside Business Analyst and Product Manager roles.

3. If you can do some part-time education, this short course from one of the ex-dbt employees looks to be worthwhile considering. It is brand new, but they intend to teach the exact thing I usually look to hire for.

4. So, onto The Analytics Engineer skills/tools to know

    I’d give 60% focus on SQL, with maybe 10% each for Git, CLI, Python, BI tools.

   1. SQL

      1. Analytical queries like this tutorial, especially the “advanced” section, also this tutorial

      2. CTEs

      3. Window functions

      4. Focus on consistency, neatness, style

   2. Git basics

      1. Pull, branch, commit, merge

      2. Most people don’t need to know much more

   3. Command-line basics

      1. Navigation, creating, deleting, moving

   4. Python

      1. Python is a huge domain, super useful but harder to get to grips with, and also less useful unless you are in a specific niche that calls for it

      2. Jupyter notebooks

      3. Basic pandas for data manipulation

      4. Functions

      5. Virtual environments basics

   5. BI Tools

      1. BI tools are often prohibitively expensive, so the experience is often limited to one, and often the wrong one or an old one.

      2. Metabase is the defacto open source quick and simple option, it functions in a reasonably useful way. It is worth downloading and testing out. 

      3. Another shoutout to lightdash, a very simple BI tool that runs with dbt, and is familiar to users of Looker.

   6. Amendment: *Excel*

      1. This post strongly assumes Excel expertise, but some readers pointed out that this would be worth stating.

      2. Pivots, xlookup etc. 

      3. Highly recommend reading the post Doing Better With Excel.

Analytics industry context:

Articles/books etc that I think worth having a look at to add some colour to the above.

1. I wrote some articles about what I think about technology, the data-as-a-utility-tool one is probably the only one worth reading.

2. Building a data team at a mid-stage startup: a short story

   1. This very neatly describes my jobs and career so far 

   2. A good representation of what the “data” industry looks like

3. The Modern Data Stack: Past, Present, and Future

   1. dbt is the tool that I use for SQL transformations. It is probably the single most useful thing to learn in addition to the SQL, git, command line, Python list

   2. They are doing lots of thought leadership in the analytics space, with very good blog posts

4. Technology in Data Analytics

   1. Simplest, shortest overview of the most important tech tools, anything not on this list is possibly out of date or redundant, or used in a different context

5. The Analytics Setup Guidebook

   1. This is a deep dive into modern analytics, opinions are now pretty generic, but it is a BOOK

   2. Some interesting meta-analysis takes on the jobs in the industry here

   3. Possibly the thing that is  done most badly in the analytics space is data modelling, with the background here. This is probably what will be the hardest to attain, and near impossible to hire for, but something that is mostly trial and error anyway!

That is all I’ve got. Topical and relevant as of publishing. I’ll continue to add and amend, but for now, if you know of someone interested in Data, send them this. There is a world of nuance not covered here, the most interesting themes all covered in due course in a post here. Or not. 

As may be obvious from the long list of articles, there are lots of thoughts and opinions in this space. I have personally experienced the tension and growth of data analytics alongside data science and data engineering, (and also a frustrated relationship with software engineering). In very many contexts the data science/engineering/analytics terms are used overly interchangeably, and often end up meaning the exact same thing, but often not meaning anything similar at all, just to make it confusing, especially data engineering. See an upcoming post on Data Engineering: Backend Developer, or Data Analyst.

And as I said, I’d love to hear from you if you are making this transition, or feel like understanding it in more detail. The above is just a primer! Get in touch to set up time directly, quickest via the dreaded: https://www.linkedin.com/in/m-ard/

Please consider subscribing for more on the subject of data systems thinking

What is group by 1

Who is Matt Arderne

Dataform and dbt, Dbt and dataform

If neither dbt nor Dataform is familiar to you, stop and read my primer post on the modern data analytics stack. This post will still likely be a bit meta unless you are familiar with at least one of these tools. The below two-liner explanation from No frills data warehousing with dbt might be sufficient to get you through.

To use [dbt/Dataform], you only need to be familiar with SQL. The package relies on templating using [jinja/javascript] to enable nifty features like dependency graphs, macros or schema tests. Upon compilation, everything is translated into pure SQL and run on the database’s execution engine. It is quite fascinating how much you can do with such a minimalist tool.

Context

Because I find it interesting, here is my understanding of the history of these tools. Cmd+f to THE TLDR if the what why how and hearsay of technology isn’t your thing.

dbt was born from a common need. Fishtown Analytics, an analytics consulting team, solved their need for a better way to transform data inside a data warehouse. They built a way to manage SQL transformations, as an open-source tool way back in 2016 (epic to git time-travel).

They generously shared their experience through a transformative series of blog posts from their CEO Tristan Handy. Edit: Here is a full history! . He captured many struggling engineer/analysts attention with his “new way” of doing analytics in startups. We, the desperate, listened closely. The message I heard: bring the best of software development to startup data analytics

This was great, but what set this apart amongst the ever-growing set of open-source developer tools was, in my opinion, the “hype house” that was the early dbt slack community.

The Hype House is part of a millennia-old tradition of collaboration among those at the avant-garde of new forms of media, technology, and thought. Outsiders like me have always dismissed the novel as silly, faddish, or worse. When those inside the cutting-edge scenes band together to support, teach, and create with each other, their niche and experimental projects can become the new normal on top of which the next generation builds.

Source (I’ve waited a long time to paste this snippet. Scenius is quite a word, but the article is epic)

The building of this dbt tool was “in the open”, and a community was incubated alongside it. An enthusiastic community. An investible community. The new paradigm was incubated in the dbt slack channel, with contribution, opinion and criticism all weighed and measured in the passionate and growing community.

dbt was first a CLI tool. As is the trend with open-source, evolution is dynamic and open. Dataform was born when a front-end was created by a team of engineers who spotted the opportunity to bring dbt to even less technical analysts through a GUI. Later on, Dataform decided to migrate to a newly built dbt replacement backend for their frontend (also open-source). To match this development, or they had planned to anyway, dbt launched their own GUI SaaS tool (called Sinter, now dbt Cloud for the historians).

Personally, dbt led me to Dataform, and as a consulting data-engineer turned head-of-data, I was drawn by Dataform’s relative ease of getting started. I wanted a tool that I could stick in the hands of a Looker developer and have them hitting the same notes as I was hitting with dbt, with less friction, less cognitive burden, and less fiddle.

So, this post. We’ll briefly run through the most notable differences between dbt and Dataform, how/why to choose, and end with some thoughts on evolution.

Why me? I’ve been working with these tools for a while. I deployed dbt at a company and then migrated to Dataform when Dataform launched. I’ve deployed Dataform with another client and recently worked on a dbt project that used some of the dbt plugins. I now consider both on their relative merits when I make the assessments below.

If you’ve got this far but can go no further, these tools function similarly, with the TLDR worth a look.

The bulk of the value for me has been in having these tools as “ready to use” for data analysts who are strong with LookML, SQL and have exposure to data modelling concepts, but are less familiar with Airflow, CLI tools, Python environments, and git workflows (git undo everything). The singular terror that it is to do anything Python-related on Windows (as a Mac user) has been reason enough to default to avoid having to set someone up with the CLI, enabling git, managing Python dependencies, C++ redistributables, :shock:. Both tools allow developers to move in that direction if they choose, but offer easier onboarding via the SaaS version.

Thus, dbt/Dataform cloud is my primary point of consideration, because that has been my primary interest. Both are extensively used as CLI tools, which for many is the go-to. My focus has been in enabling teams that initially do not have the time/resources to upskill/maintain/support the tech, and so my considerations are within the context of the cloud offering. This may be contentious, but on average the analysts who come from Bizops or something of that nature seem to make for more rounded data analysts, and enabling them is key.

*complete aside, Denodo, a previous employer and data integration technology company, was born out of the need of their consulting team working with large multinationals. They too built a successful enterprise Data Virtualisation tool, which in a way achieves a similar outcome to dbt + Snowflake. In essence a powerful enterprise database multi-plug adaptor, and a SQL DAG builder.

dbt

dbt is an open-source tech success story. A dedicated team created the technology they wished they could pay for, built in the open, engaging the community and incubating a “new way” that enthusiasts began to feel very enthusiastic about.

dbt’s strengths lie in the powerful ecosystem they have developed. Illustrative of this are the integrations with tools such as Fivetran, Census, Databricks, that make raw dbt projects pretty mobile and feel like the beginnings of a standard for SQL transformation DAGs.

This ecosystem enables developers to build powerful open-source plugins, available to use and improve. These include useful macros, prebuilt transformations for specific sources, infrastructure management and even some interesting machine learning plugins. The community has built the tools they need on a common standard.

Because dbt is platform-agnostic it feels like an ecosystem rather than a tool alone. Any modern data warehouse is a viable engine to run it, and any modern data tool is likely considering how they can incorporate elements of it to take advantage of the momentum.

When considering dbt cloud, the online version it feels like an online IDE rather than a standalone tool, and according to the founder as of a 1 year ago they’re

in the very early days in improving the developer experience of writing dbt code. The dbt Cloud IDE is still in its infancy, and is only one of the many ways in which we ultimately believe that users will write dbt code that we want to facilitate. source

My take is that this rings true. The usability features are not quite there, a simple example is no autocomplete, which readily exists as a VSCode plugin. [TODO: Validate this is still true (please comment below)]

Dataform

Dataform built a great user interface and my initial impression was that it felt like an easy transition for someone from Looker to orientate and become productive quickly. The user interface has continuously been improved and the backlog of feature requests was quickly moved through, as the team was reactive to suggestions and rapidly added features that provided better prompts, insights and contextual information. Things like prompting about an unbuilt view being referenced, an effective autocomplete, and continuously compiling the SQL to raise errors like typos, glitches, invalid elements introduced prior to execution. These all added to the great user experience of developing.

My least favourite feature is that any templating is done in Javascript (instead of jinja in dbt). Ideally, I’d prefer something closer to Python than either, but jinja feels easier and more intuitive to a non .js type. 

[Ammendment] A friend’s opinion, below, which runs counter to mine on the jinja/js thing, and I agree with him, especially if you have a team who know Javascript! My counterpoint is that Looker analysts often find jinja easier to get started. Jinja does get very complicated if you try and do things like cohort funnel analysis.

Your main disadvantage of dataform I actually found to be a great advantage: templating is just javascript. Sometimes jinja starts feeling like a programming language (macros in macros etc.). But that makes a horrible developer experience. In dataform, you can just declare models in a regular old .js file, meaning you’re completely free to build whatever you want. Plus you have all the power and tooling of a mature programming language.

[End Ammendment]

Like dbt, Dataform has benefitted from community-maintained packages.

The most interesting thing about this story is that Dataform was acquired by Google. It seems likely that Dataform will be incorporated into Google’s Big Query data warehouse, I suppose as a “transformation” feature, or perhaps still standalone? This has interesting implications but it does mean that there will be some time getting Dataform integrated. It also means that as a tool it may have less of their historic great responsiveness to user suggestions and demands as they run through the enterprise compliance backlog. This does mean that it will be an easy first choice for Big Query customers to test out this paradigm of data transformation. Sadly, it also means that support for other data warehouses has been sunset.

Implications

The implications of the Google acquisition of Dataform are interesting to me, maybe given my unique experience with both tools, in that it pretty firmly plants dbt as the SQL data transformation standard. This has likely been the case regardless, based purely on momentum and head start that dbt has maintained. 

However, had Google maintained Dataform as an independent entity rather than incorporating it into Big Query as an exclusive feature or tool, it may have given Dataform the staging area and resources to properly pose a serious threat as an alternative. This would have been a preferable outcome for me personally. Dataform going exclusively Big Query suggests that the play is likely a more direct threat towards Snowflake’s dominance in the data warehouse space, as the feature race heats up.

From a more practical perspective, the TLDR comes into play. When heading along this road, the first decision you’re likely to make is Big Query or Snowflake (or even Redshift if you have lots of AWS credits… (those are ominous dot dots)) decision, and then if Big Query, deciding between dbt and Dataform. My preference is due to my familiarity with Snowflake, and as a practitioner and consultant, my bet is generally with the coverage and standard-setting dbt. 

TLDR

THE TLDR/

• Dataform was acquired by Google and is now exclusively GCP, and my guess is that it will be integrating directly into the Big Query platform. If you’re a startup in the GCP ecosystem, then that is compelling. If you like or are already using Snowflake, then Dataform is no longer an option.

• dbt has a huge ecosystem built around it, with real momentum benefitting from plugins, integrations and general compatibility across all the analytics tools.

• Both tools have a SaaS offering, effectively a cloud IDE + CICD. Great for getting connected and developing quickly. Both happily run a hybrid of cloud IDE and open-source core.

• The dbt community has led to many ecosystem “gap fillers” such as spectacles, lightdash, and technically even Dataform itself, all taking the dbt base and extending it or recombining it in different ways.

The modern ELT data stack that no one is going to second guess has Snowflake as the data warehouse, Fivetran doing the ExtractLoad and dbt doing the T for transforms. Note that Snowflake and Fivetran are doing the heavy-lifting, but don’t provide the magic. They are doing the quiet and dependable, probably get taken for granted (other than the price tag), but developing is where insight is captured, and developing is done with dbt.

/ THE TLDR

Evolution

If you jumped to the TLDR and then went further, this is likely where you’d want to stop. The rest is a quick self-indulgent attempt at hypotho-philosophising the evolutionary aspects of software building. Beware!

Watching the evolution of dbt and then Dataform quite closely and with entrenched interest, I have found the dynamics at play interesting, engaging, and commentable. What was also interesting is the entrenched alliances that quickly developed along with people’s preferences or opportunities. Sentiment generally went something along the lines of Dataform ripped the idea off dbt, which I feel is mostly wrong, as the idea isn’t novel, just very well executed and well-timed (IMO, comment below).

I made a meta-take comment on the sentiment in another wonderful slack channel, the comment has now been repurposed for this section, as rambly and raw as it is, I thought captured some of the point:

The tribalism vibe on this was always a weird one for me. us/them always felt counter-productive and tribal [1], but then perhaps that was inevitable. Early dbt felt very much like the beginnings of a tribe: lots of in-jokes, data-eology, new/old. Exciting! 

Dataform beginning public life as a dbt front-end ~early 2018 [2] felt pretty confirming.

It did two things for (us) users: 

1. affirmed the dbt value prop [3] 

2. made using the "new-paradigm" much more accessible to non-techy-data-devs [4] 

For reasons, Dataform built their own backend dbt replacement (Google software engineers, I don't know, but they probably do). This change positioned them at odds with one another, but besides that point, the 2 different tools enabled more users to adopt the new paradigm[5].

For me, as a user of dbt and then early dataform, best enabling this new paradigm was a combination of the two, and it was/is great! Collectively this is all a quantum leap for SQL analysts. I'm beyond thrilled that I don't have to use the tools I had to use pre-dbt. 

My post-ramble point: I think evolutionary pressure on technology is bloody brilliant for users. Competition is great, iteration should be encouraged. I'm looking forward to someone building an iteration on Looker to this same end.

The evolutionary pressure did two interesting things:

1. It affirmed the need for a SaaS tool for smaller and larger teams. dbt may never have needed to create a user interface, which would have frustrated my team building analytics efforts in smaller tech communities.

2. It created _an alternative_, which is the most important way to add creativity to the space. In real-time users expressed their preferences for different ways of achieving the same goal, in one great experiment.

Interesting parallels to this comparison can loosely be considered in other data tools such as Fivetran & Stitch (closed vs open-source), Snowflake vs Redshift (developer-friendly vs cheaper on paper) and even Tableau vs Looker (old way vs new way).

Both dbt and Dataform now have relatively independent niches, having benefited from the existence of the other. I'm personally glad for having met and got to know the Dataform team and enjoyed being tangentially involved in their growth. The product is great and has been a pleasure to use, and they are nice people who have put a lot of thought into solving my team’s problems.

I think dbt's success in raising lots of VC money is great, they have created many hours of additional output for each hour of development, and more broadly for an enhancement on the thinking in this space, also great people to deal with!

---comment footnotes---

[1] - Tribal ie The Robbers Cave Study

The Robbers Cave experiment studied how hostilities quickly developed between two groups of boys at a summer camp. The researchers were later able to reduce the tensions between the two groups by having them work towards shared goals. The Robbers Cave study helps to illustrate several key ideas in psychology, including realistic conflict theory, social identity theory, and the contact hypothesis. Source (disclaimer - I read about robbers cave this week)

[2] - dataform announcement date ~ May 2018

[3] - dbt on dataform

Hah! This is one of the implications of open-source! ... In my 3-year post, I explicitly called out this likelihood and embraced it

[4] - early dbt-cloud called Sinter, early 2017

[5] See superordinate goals conveniently linked in [1]

Closing

I enjoyed witnessing first-hand the evolution of "a new technology". But I think it probably fairer to call it a recombination, or a reorg. Bottoms up top-down shuffle.

The reshuffle rather than blinding innovation here is made clear by the fact that I could port the code I wrote in 2016 in Denodo’s VQL query language, designed in 2008, which was a large and involved SQL DAG, into something that dbt and Snowflake would happily use. Entirely independently from dbt, and I’d bet mutually unknown, a company created from a users perspective in essence the same way of working. I’m sure from an evolutionary perspective this divergent and now convergent evolution is an interesting quirk.

In that sense, nothing here is “new” enough to interest a computer scientist. The computer scientists I know have an allergic reaction to most of the analytics tooling. Messy and inefficient new ways of doing things, which then become the standards and that look like inflexion points after the fact, and I think for an analyst this change is in progress. Going back to the hype house concept:

Outsiders … always dismissed the novel as silly, faddish, or worse. When those inside the cutting-edge scenes band together to support, teach, and create with each other, their niche and experimental projects can become the new normal

The user is one of the many pressures that drive the evolutionary development of these tools but is best served by choice. Consolidation of technology into profit mode leaves the user’s needs as a low priority, whereas competitive pressure and alternatives put the users at the front. Current trends are about enabling a data analyst turned developer to create as much insight with as little engineering as possible, making it a great time to be an analyst. The users ultimately benefit when the technology scene is in inclusive, collaborative and open mode. Ultimately we are probably due for a period of consolidation mode, but perhaps more on that later.

Please comment if you have any feedback on any of this, I aim to improve with your help.

* again interesting aside: Denodo was part of a different paradigm, with virtualisation being the word. Instead of ETL’ing your data into a data warehouse, you’d rather leave it where it lives and query it directly. Whether in a data warehouse, API, transaction database, or many others! A dream for a data architect faced with decades of legacy tech and a desperate need to unify access for analytics and as a data bus. Dremio is now continuing this trend, focussing more on the data lake concept.

Please consider subscribing for more on the subject of data systems thinking

What is group by 1

Who is Matt Arderne