Three teams. Three repositories.
One lineage.

The build, live

The stitched graph of the three-repository run. Click any column: its provenance lights up in both directions, across every engine and every repository it passes through. Each table is labelled with the repository that owns it.

Why one project does not scale

Three reasons, none of them about dbt.

Large regulated organisations tend to land on a hub-and-spoke arrangement rather than a textbook data mesh: a central platform team owns the catalog, the standards and the evidence; domain teams own their models, tests and release cadence in their own repositories. The shape below is that arrangement. The estate file is the hub. The team repositories are the spokes. The lineage is the evidence.

What changed

One project became four repositories. Nothing in the model SQL changed; only the namespaces carry a suffix so the two shapes can coexist and be compared.

The shape, and where the next team plugs in

Hub and spoke. The hub holds no model code, no data and no compute; it holds the list of projects and the evidence. Every team depends on others only through declared identities, drawn in blue.

What it cost to add a team

Scalability is a claim about the marginal team, not the first three. This is what adding the third team, the ML team on a different engine in a different cloud, touched in our build, and what it did not.

Not touched

• —The producing team’s repository: 0 files changed.
• —The analytics team’s repository: 0 files changed.
• —The stitcher and the system tables: 0 changes for the new team (the estate-mode change itself was made once, before).
• —The shared copy of the data: nothing moved, nothing duplicated.

We have built three teams, not thirty. What the table shows is the mechanism: a team joins by adding, never by editing another team’s repository. Whether an organisation’s review, release and access processes keep that property is up to the organisation, and is not something this build can show.

The contract, in two files

Teams share exactly two things. A consumer declares what it reads and which physical table that is. The central team declares which projects exist and in what order they depend on each other. Neither file contains model code.

The consumer’s side: declared sources with a physical identity

This is the ML team’s repository. It depends on two other teams and names, for each table, the engine-independent identity that the producing team’s model resolves to. This is the same mechanism the single project used; the difference is that it now crosses a repository boundary as well as an engine boundary.

sources:
  - name: stock_common_dbx              # produced by the Athena team, in their repository
    database: "{{ var('dbx_bridge_catalog') }}"
    schema: stock_common_mesh
    tables:
      - name: common_price_daily
        meta: { physical_identity: "glue:stock_common_mesh.common_price_daily" }
      - name: common_ticker
        meta: { physical_identity: "glue:stock_common_mesh.common_ticker" }
  - name: stock_consumer_sf             # produced by the Snowflake team, in their repository
    database: "{{ var('dbx_bridge_catalog') }}"
    schema: stock_consumer_mesh
    tables:
      - name: mart_price_monthly_usd
        meta: { physical_identity: "snowflake:datapai.stock_consumer_mesh.mart_price_monthly_usd" }

The central team’s side: the estate file

Which projects exist, which engine builds each, the catalog sync a producer needs so other engines can find its tables, the read grants a consumer engine needs, and the pipeline under which the stitched lineage is stored. Adding a fifth team is a line in this file.

pipeline_id: stock_xengine_mesh

snowflake_sync:                        # producer-owned Glue Iceberg -> read-only pointers in Snowflake (once per new table)
  - { glue_database: stock_common_mesh, database: DATAPAI, schema: STOCK_COMMON_MESH,
      mode: glue, glue_catalog_integration: GLUE_STOCK_COMMON, external_volume: DATAPAI_S3_VOL }

reader_grants:                         # what a consumer engine's reader role must be allowed to see
  - { roles: [DATABRICKS_READER], schemas: [DATAPAI.STOCK_COMMON_MESH, DATAPAI.STOCK_CONSUMER_MESH] }

projects:                              # producer first, then consumers; each entry = one repository's dbt artefacts
  - { name: xengine-producer-stock,     project_dir: ../xengine-producer-stock,     engine: athena,     dialect: athena }
  - { name: xengine-consumer-analytics, project_dir: ../xengine-consumer-analytics, engine: snowflake,  dialect: snowflake }
  - { name: xengine-consumer-ml,        project_dir: ../xengine-consumer-ml,        engine: databricks, dialect: databricks }

What the stitcher had to learn

Very little, which is the point. It already joined per-engine graphs on the declared identity. The change was to let each entry bring its own project directory, and to record which project each table came from. The join itself never read project membership, so nothing about it changed.

for e in engines:                                   # producer -> consumers
    pdir  = e.get("project_dir", project_dir)         # NEW: this entry's own repository (default: the single project)
    pname = e.get("name")
    man   = manifest(pdir, e["target_dir"])
    cat   = catalog(pdir, e["target_dir"])["nodes"]
    lin   = lineage_from_artifacts(pdir, dialect=e["dialect"], target_dir=e["target_dir"])
    ...
    nodes[n["name"]] = {"engine": eng, "project": pname, "identity": ident,   # NEW: project recorded on every table
                        "built": bool(cat.get(n["unique_id"], {}).get("columns"))}

Running it

xengine-producer-stock/      dbt build ; dbt docs generate
xengine snowflake-sync --estate estate.yml       # once per new producer table; auto-refresh follows afterwards
xengine grant-readers  --estate estate.yml       # once per new schema a consumer's reader role must see
xengine-consumer-analytics/  dbt build ; dbt docs generate
xengine-consumer-ml/         dbt build ; dbt docs generate
xengine lineage --estate estate.yml --to-db      # one graph across the repositories -> system tables

What we measured

Both shapes were built from the same raw data on 22 September 2026, in our demo environment, and compared table by table. Correctness only; no timings.

Two value checks as well. The latest month-end close of one ticker in the Snowflake mart was identical to the last digit in both shapes. The Databricks momentum features were joined on their key across the two shapes: 8,085,051 rows compared, 0 differing beyond 1e-9.

What this establishes: the lineage join is on physical identity, not on project membership. Splitting the estate into repositories owned by different teams, on different engines, in two clouds, changed nothing in the graph or in the data. Before this run we could argue that from the code; now it is a measurement.

Three things that broke

All three are about permissions and order, not about lineage, which is what we would expect the first day a second team appears.

What this does not show

• —Two tenants of the same engine. Identities are catalog-qualified (glue:<database>.<table>, snowflake:<database>.<schema>.<table>); two accounts with the same database name would collide. Account-qualified identities are the next change, and the multi-tenant claim waits for it.
• —Team autonomy beyond the repository boundary: separate CI, separate release approval and separate credentials management were not exercised. Each repository was built by hand, in order, from one machine.
• —Contract enforcement. A consumer declares what it reads; nothing yet stops a producer from renaming a column. The lineage would show the break; it would not prevent it.
• —The model SQL exists twice, once in the single project and once across the team repositories, so that the two shapes could be compared. That is a test arrangement, not a recommendation.
• —Storage on Azure Data Lake Storage or Google Cloud Storage, or a catalog other than Glue: not exercised. The producer wrote to S3 and Glue in this run.
• —Scale, cost and timings. Sample data, small compute, our own accounts. Not a benchmark.