Architecture

Agent-Orchestrated Data Pipelines: Building the Dagster + dbt + MCP Stack

By IdeaBosqueLast updated: August 28, 2026

Key takeaways

Fivetran and dbt Labs completed a merger on June 1, 2026 (~$600M combined ARR, 100,000+ data teams) and shipped Agents Schema — an open standard that turns a warehouse schema into the governed shared-context layer for agents.
Databricks reports over 80% of the databases on its Neon unit are provisioned by AI agents, not humans — the primary consumer of the data stack has already shifted from the analyst to the agent.
Three agentic patterns define the new pipeline — agents write and scaffold pipeline code, pipelines self-heal by proposing and applying fixes, and agents triage run failures in chat — each of which needs machine-readable lineage, not a rendered dashboard.
The stack is three governed layers — Dagster for asset lineage, dbt for tested transformations and shared context, and an MCP module for policy-scoped access — so an agent reads the pipeline's structure under the same controls a human would.

The data stack was built for human analysts: run the pipeline overnight, read a dashboard in the morning, file a ticket when a number looks wrong. AI agents consume data differently. As the merged Fivetran + dbt Labs put it, agents "operate continuously, in parallel, and at machine speed" — and they need the pipeline's structure (lineage, tests, definitions), not just its output. In one quarter, the largest data-movement and transformation vendors rebuilt around that fact: the Fivetran + dbt merger shipped Agents Schema and open-sourced the dbt Fusion engine as dbt Core v2.0, and Databricks acquired Electric to give each agent its own disposable Postgres.

This guide builds the pattern in a B2B procurement context: a pipeline that ingests supplier catalogs, prices, and inventory, and exposes them to an RFQ agent. It covers the three layers — Dagster for asset-centric orchestration, dbt for governed transformation, and an MCP module for scoped agent access — and the three agentic patterns that make the pipeline maintain itself. You will end knowing what each layer contributes, why asset lineage is the load-bearing choice, and where the human stays in the loop.

Why asset-centric orchestration is the foundation

Most orchestration failures for agents start with the wrong mental model. Task-centric schedulers (the classic cron-plus-DAG design) answer "did this job run?" An agent asking "why is the Acme price stale?" needs a different answer: "which data asset is out of date, what does it depend on, and what feeds it?" That is an asset question, and it is why Dagster's asset-centric model is the foundation here rather than a preference.

In Dagster you declare the thing you produce — supplier_catalog, normalized_prices, availability_snapshot — and its dependencies. The orchestrator then knows the full lineage graph. Dagster's Declarative Automation lets an asset refresh when its upstream changes rather than on a fixed clock, so "stale" becomes a property the system can reason about. That lineage graph is exactly what an agent needs to trace a bad number back to its source without guessing.

import dagster as dg

@dg.asset(group_name="procurement")
def supplier_catalog(context: dg.AssetExecutionContext) -> dg.MaterializeResult:
    rows = fetch_supplier_feed()  # NetSuite, EDI, CSV drop, etc.
    write_bronze("supplier_catalog", rows)
    return dg.MaterializeResult(metadata={"row_count": len(rows)})

@dg.asset(deps=[supplier_catalog], group_name="procurement",
          automation_condition=dg.AutomationCondition.eager())
def normalized_prices() -> None:
    # dbt owns the transformation logic; Dagster owns the lineage + trigger
    run_dbt(select="normalized_prices")

The deps and automation_condition are the whole point: the agent (and the self-healing loop below) can read this graph as data. Airflow reached the same conclusion from the other direction — Airflow 3.2 added a Common AI Provider and asset-aware scheduling — and the consolidation is real: Prefect acquired Dagster in July 2026. Whichever orchestrator you standardize on, the requirement is the same: assets with declared lineage, not opaque tasks.

Why dbt owns transformation and the governed context

Dagster triggers work and tracks lineage; it should not contain your business logic. That belongs in dbt, where every transformation is a version-controlled SQL model with tests, documentation, and a semantic definition attached. For agents this is not a nicety — it is the trust boundary. dbt's own position is that the transformation layer is what makes agentic pipelines trustworthy: an agent that writes SQL against undefined, untested tables automates chaos faster.

The post-merger addition that matters most is Agents Schema: a designated warehouse schema that stores metric definitions, semantic models, dbt lineage, and business documentation as plain SQL tables. Instead of every agent re-deriving what "on-hand availability" means, the definition lives in one governed, customer-owned place the agent reads from. It is the data-side counterpart to a governed connector module — a single, policy-scoped source of shared context rather than a per-agent copy that drifts.

-- models/marts/availability_snapshot.sql
select
    sku,
    warehouse_id,
    on_hand - allocated as available_qty,   -- the governed definition
    updated_at
from {{ ref('normalized_inventory') }}

-- schema.yml: the test that gates the agent's trust
-- - name: available_qty
--   tests: [not_null, {dbt_utils.accepted_range: {min_value: 0}}]

A test that fails is a signal the agent should not quote against that row. That single fact — a machine-readable pass/fail on every model — is what lets the next two patterns run without a human watching every step.

Where the agent connects: an MCP module, not a database login

An agent should never hold raw warehouse credentials. It should call a governed MCP module that exposes a small set of typed tools — get_availability(sku, warehouse), get_tier_price(sku, customer_tier), list_substitutes(sku) — each mapping to a tested dbt model and each carrying policy scope, rate limits, and audit logging. This is the same module pattern used for ERP and commerce connectors, applied to the pipeline's own output. It keeps the blast radius small: the agent can read availability_snapshot but cannot run arbitrary SQL, and every call is logged.

That boundary is also where per-agent state fits. Databricks' acquisition of Electric — WASM Postgres (PGlite) inside the agent sandbox, synced to central governed state — exists because agents "need thousands of tiny, disposable databases" for working context, kept separate from the durable, governed tables. The rule of thumb: durable, shared, governed data lives behind the MCP module; fast-moving, per-run scratch context lives in the agent's own sandbox.

The three agentic patterns this stack enables

With lineage (Dagster), tested definitions and shared context (dbt + Agents Schema), and scoped access (MCP) in place, three patterns become practical:

Agentic development. Agents scaffold new assets and transformations — draft the dbt model, propose the schema test, wire the Dagster asset — against the existing lineage graph. Dagster ships dagster-io/skills for Claude Code and Codex and a Compass Slack assistant; Bruin exposes an MCP server for the same purpose. The human reviews a pull request, not a blank file.
Self-healing pipelines. When a schema test fails or an upstream asset breaks, the agent reads the lineage, isolates the failing model, proposes a fix, and either applies it in a canary run or files a PR. Because Dagster knows the dependency graph and dbt knows which test failed, the fix is lineage-aware rather than a blind retry.
Agentic troubleshooting. On failure, the agent reads the run logs and metadata and responds in Slack or Teams with the likely cause and a proposed fix — the pattern Dagster Compass and Snowflake Cortex are built around. On-call debugging shifts from reading dashboards to reviewing an agent's diagnosis.

None of these removes the human. Every applied change passes a test gate, a canary, or a review — the same discipline the dbt Summit 2026 keynotes framed as the price of letting agents touch production data.

The stack, in one view

A representative build

A distributor running NetSuite, two warehouses, and three supplier catalogs wanted an RFQ agent that could quote without a human pulling availability by hand. The pipeline was the blocker, not the model. We declared the catalog, price, and inventory assets in Dagster with explicit lineage; moved the pricing and availability logic into tested dbt models with an Agents Schema that pinned the definition of "available to promise"; and exposed three typed tools through an MCP module scoped to read-only. The self-healing loop now catches a broken supplier feed and files a PR with the fix before the morning quote run; on-call time on pipeline breakage dropped because the agent's first response is a diagnosis, not a page. The agent quotes against tested data or refuses — it never quotes against a row that failed its test.