Decouple your data from your engine.
The engineering behind it.

The build, live

Click any column. Its provenance lights up in both directions, across every engine it passes through.

Architecture

The same models run on three engines. Each engine writes to its own catalog. The shared tables and the Snowflake marts are one set of Iceberg files in one S3 bucket, read by every engine. Databricks writes its own feature tables to its own Azure storage.

Who writes, who reads, through which catalog

This table is the core of the design. Open files were necessary but not sufficient for us; which catalog owns a table decided which engines could find it.

How it is put together

Six steps. It uses dbt-core, an open table format, catalog features the engines document publicly, and our own code for stitching the lineage.

1. 01

   One environment, three adapters

   dbt-core 1.9 with the Snowflake, Athena and Databricks adapters in a single Python environment. One profiles file; each engine is a target of one profile, and the connection definitions are reused, not copied.

2. 02

   Folders switch on by engine — the single-project shortcut

   We put producer and consumers in one dbt project and enable model and source folders by target type, so each engine sees only its own slice. Source names must be unique project-wide, even when disabled. One project is the simplest shape for a demonstration, not the shape we would recommend for many teams; see the note at the end of this section.

3. 03

   Producer publishes to an open catalog

   Athena builds the shared tables as Iceberg into Glue, as merge models with a unique storage location per table.

4. 04

   Consumers attach without copying

   Snowflake: a Glue catalog integration plus the existing external volume. Databricks on Azure: a read-only cross-cloud storage credential, an external location, and one foreign catalog over the Snowflake database.

5. 05

   Lineage per engine, then stitched

   Each target writes its own manifest and catalog. SQL is parsed per dialect; graphs are joined where a consumer source and a producer model declare the same physical identity.

6. 06

   Persist and serve

   Edges and tables go to plain system tables with the engine on both sides of every edge, and to one JSON file that the lineage page reads.

The project file: folders switch on by engine

models:
  stock_xengine:
    +tags: ["cross_engine_producer_consumer", "cross_engine"]
    producer_athena:
      +enabled: "{{ target.type == 'athena' }}"
      +tags: ["role_producer", "engine_athena"]
      +table_type: iceberg
    consumer_snowflake:
      +enabled: "{{ target.type == 'snowflake' }}"
      +tags: ["role_consumer", "engine_snowflake"]
      +table_format: iceberg
    consumer_databricks:
      +enabled: "{{ target.type == 'databricks' }}"
      +tags: ["role_consumer", "engine_databricks"]

sources:            # switch sources per engine too, or each engine tries to catalogue the others'
  stock_xengine:
    producer_athena:     { +enabled: "{{ target.type == 'athena' }}" }
    consumer_snowflake:  { +enabled: "{{ target.type == 'snowflake' }}" }
    consumer_databricks: { +enabled: "{{ target.type == 'databricks' }}" }

Shared tables are merge models, on purpose

In our setup a rebuilt table got a new Iceberg identity on every run, while a merged table kept one and added snapshots. The consuming engine could follow the second, not the first.

{{ config(
    materialized         = 'incremental',
    incremental_strategy = 'merge',
    unique_key           = ['ticker', 'exchange', 'trade_date'],
    on_schema_change     = 'append_new_columns'
) }}
-- every run re-merges the last 7 days; one table identity, new snapshots

Where the lineage graphs join

dbt gives one manifest per target, so each engine’s graph ends at a source. The consumer declares which physical table that source is; the producer’s model resolves to the same identity.

sources:
  - name: stock_common_dbx              # produced by Athena; Databricks is the second consumer
    database: "{{ var('dbx_bridge_catalog') }}"
    schema: stock_common
    tables:
      - name: common_price_daily
        meta: { physical_identity: "glue:stock_common.common_price_daily" }
  - name: stock_consumer_sf             # produced by Snowflake, read from the same Iceberg files
    schema: stock_consumer
    tables:
      - name: mart_price_monthly_usd
        meta: { physical_identity: "snowflake:datapai.stock_consumer.mart_price_monthly_usd" }

Running it

dbt build --target athena     --select tag:role_producer,tag:engine_athena
dbt build --target snowflake  --select tag:role_consumer,tag:engine_snowflake
dbt build --target databricks --select tag:role_consumer,tag:engine_databricks

# artefacts per engine, then one stitched graph -> JSON for the page + system tables
dbt docs generate --target <engine> --target-path target_<engine>
xengine lineage --project . --to-db

The auditor’s question, in SQL

Which numbers left one engine for another? Every edge stores the engine on both sides.

select from_engine || ' -> ' || to_engine as crossing, count(*) as column_edges
from   v_sys_ai_column_lineage_latest
where  pipeline_id = 'stock_xengine'
group  by 1 order by 1;

 athena    -> athena        29
 athena    -> databricks    16
 athena    -> snowflake     24
 snowflake -> databricks     6

The part we had to build

Everything above is configuration of open tools and vendor features. It gets you three engines over one copy, and three separate lineage graphs. Four pieces of our own code turn that into one graph with evidence in your database. Excerpts from the working repository, lightly trimmed.

1 · Stitching the per-engine graphs

dbt produces one manifest per target. Each engine’s graph therefore ends at a source, and no engine knows that its source is another engine’s model. The stitcher walks the engines in producer-to-consumer order, parses every model’s compiled SQL in that engine’s dialect for column edges, and when a source declares a physical identity that an earlier engine’s model already claimed, it aliases the source onto that model. A model is marked built only if the engine’s own catalog lists its relation with columns; otherwise the lineage is of code, not of a table.

for e in engines:                                   # producer -> consumers
    man = manifest(project_dir, e["target_dir"])
    cat = catalog(project_dir, e["target_dir"])["nodes"]
    lin = lineage_from_artifacts(project_dir, dialect=e["dialect"], target_dir=e["target_dir"])

    source_alias = {}                              # "source:<name>" in THIS engine -> stitched node
    for s in man["sources"].values():
        ident = s["meta"].get("physical_identity")  # e.g. glue:stock_common.common_price_daily
        if ident in identity_to_node:               # an upstream engine PRODUCED this table
            source_alias[f"source:{s['name']}"] = identity_to_node[ident]
        else:
            nodes[f"source:{s['name']}"] = {"engine": eng, "role": "source", "identity": ident}

    for n in models(man):
        ident = model_identity(n, eng)              # glue:<db>.<t> | snowflake:<db>.<schema>.<t>
        nodes[n["name"]] = {"engine": eng, "identity": ident,
                            "built": bool(cat.get(n["unique_id"], {}).get("columns"))}
        identity_to_node[ident] = n["name"]

    for ed in lin["edges"]:                         # column -> column, from the parsed SQL
        frm = source_alias.get(ed["from_node"], ed["from_node"])
        edges.append({**ed, "from_node": frm, "to_engine": eng,
                      "from_engine": nodes[frm]["engine"]})

crossings = [e for e in edges if e["from_engine"] != e["to_engine"]]

The result on this project: 12 tables, 75 column edges, 46 of them crossing an engine boundary, 0 unresolved, 0 not built. Those four numbers are printed by the tool and are the ones we would want a reader to check first.

2 · Keeping the catalogs in step

Open files do not announce themselves. An engine finds a table through its catalog, and each engine has its own. Two small tools carry the current Iceberg metadata pointer across. Snowflake to Glue asks Snowflake where the table’s metadata file is and registers a Glue table pointing at it. Glue to Snowflake creates a read-only Iceberg table in Snowflake that follows the Glue entry, so a producer rewrite is picked up without a re-sync.

cur.execute(f"show iceberg tables in schema {database}.{schema}")
for t in cur.fetchall():
    cur.execute(f"select system$get_iceberg_table_information('{database}.{schema}.{t.name}')")
    meta = json.loads(cur.fetchone()[0])["metadataLocation"]   # s3://.../metadata/00012-....metadata.json
    glue.update_table(DatabaseName=glue_db, TableInput={
        "Name": t.name.lower(), "TableType": "EXTERNAL_TABLE",
        "Parameters": {"table_type": "ICEBERG", "metadata_location": meta, "written_by": "snowflake"},
        "StorageDescriptor": {"Columns": columns_from_iceberg_metadata(meta), "Location": meta.rsplit("/metadata/", 1)[0]}})

# catalog_namespace = the table's own Glue database, so ONE integration serves every database.
# CREATE OR REPLACE swaps a read-only pointer; the producer owns the files, nothing is deleted.
cur.execute(f"""
  create or replace iceberg table {db}.{schema}.{name}
    external_volume  = '{external_volume}'
    catalog          = '{glue_catalog_integration}'
    catalog_namespace  = '{glue_database}'
    catalog_table_name = '{name}'
    auto_refresh = true
""")

3 · Evidence as tables in your own database

The stitched graph is not kept in a tool. Every edge is written with the engine on both sides and the engine-independent identity it was joined on; every table with whether it was built. The migration is additive: single-engine lineage already in the tables keeps working with NULLs in the new columns.

alter table datapai.sys_ai_column_lineage
    add column if not exists from_engine   text,   -- engine that BUILDS from_node
    add column if not exists to_engine     text,
    add column if not exists from_identity text,   -- glue:stock_common.common_price_daily
    add column if not exists to_identity   text;   -- snowflake:datapai.stock_consumer.mart_x

create table if not exists datapai.sys_ai_lineage_node (
    pipeline_id text not null, run_id text not null, run_at timestamptz not null default now(),
    node text not null, engine text not null, role text not null,      -- source | producer | consumer
    identity text, relation text,
    built boolean,          -- true: relation exists in the engine catalog; false: lineage from code only
    unique (pipeline_id, run_id, node));

create or replace view datapai.v_sys_ai_column_lineage_cross_engine as
select pipeline_id, from_engine, from_node, from_column, to_engine, to_node, to_column, from_identity
from   datapai.v_sys_ai_column_lineage_latest
where  from_engine <> to_engine;               -- the auditor's question, as a view

4 · The connection kits

Connecting three engines across two clouds took an IAM role with two trust relationships, a Glue-read policy that needed one permission the documentation did not make obvious, a Snowflake catalog integration, a Snowflake reader role with a key pair, a Unity Catalog storage credential, external location, connection and foreign catalog. Each of these was done by hand once, then turned into a template that renders the exact statements for a given environment, plus a script that captures the current definition so it can be rolled back. That is the difference between a demo that worked once and a step that can be repeated in someone else’s estate.

snowflake_glue_integration/
  render.py --from-snowflake        # reads the integration's IAM user/external-id, renders trust + read policies
  params.datapai.yml                # account, role name, glue databases, region
databricks_azure_reads_snowflake_iceberg/
  1_s3_read_policy.json.tmpl         2_trust_policy.json.tmpl
  3_snowflake_reader_role.sql.tmpl   4_databricks_federation.sql.tmpl
  capture_foreign_catalog.py        # DESCRIBE -> re-creatable DDL, incl. storage_root, before any drop
  rollback/                         # what the capture produced for our own environment

In one table

The layer owns no compute and no storage. It sits beside the engines, which is why it can follow data across all of them, and why it can be switched off without moving any data: the Iceberg files, the catalogs and the evidence tables all remain where they are. The same code base generates the dbt models, their tests and a personal-data map from a source schema on a single engine today (Data Governance); carrying the personal-data flags across the engine boundary is the next item on the roadmap below.

What we measured

Same data, three engines. These are correctness checks from our demo environment on 22 September 2026, using sample market data. They are not a benchmark and not a comparison of engines; we deliberately publish no timings.

Five things that broke

The useful part. These are things we observed in our own build. They are notes on our configuration, not statements about how any product behaves in general.

Strategy and roadmap

The build is one step in a direction: take back, layer by layer, what has become bound to a vendor. Three horizons.

How a regulated enterprise could adopt it, without a big-bang

A path, not a promise. Each step stands on its own and can be stopped after; none requires committing to the next.

1. 01

   Start with evidence, not migration

   Point the lineage at one existing domain, read-only. Nothing moves. The first deliverable is an answer to “where did this number come from” that the business did not have before.

2. 02

   Put one shared domain on an open format

   Choose a domain several teams consume. Publish it once, as merge tables owned in an open catalog. Existing consumers keep working; new ones attach without a copy.

3. 03

   Prove an exit on one workload

   Build the same model on a second engine over the same files and compare the results. That comparison is exit-readiness evidence a risk committee can read — before anyone decides to move anything.

4. 04

   Place workloads deliberately

   With lineage and tests following the model, decide per workload where it should run, and revisit the decision when prices or needs change.

5. 05

   Extend the same control to AI

   Bring models inside the same boundary, gated by the classification the pipelines already produce.

What this does not claim

• “Data sovereignty” here means ownership and independence. Your data in your storage, in an open format, with governance in your own tables, so engines are replaceable. The build runs on public cloud services; legal jurisdiction, encryption-key control and who operates the platform are separate questions it does not address.
• This is a demonstration, not a production reference. One project, sample data, small compute, our own accounts. It shows that the approach works end to end; it does not show how it behaves at your scale.
• Not every engine reads every file directly. Databricks in Azure reads the Snowflake-produced tables straight from the Iceberg files. It reads the Athena-produced tables through Snowflake, because when we built this (September 2026) AWS Glue was not among the documented catalog federation sources for Azure Databricks. The results are identical; the route is not. Vendor capabilities change, so check current documentation.
• Not every table is Iceberg. The shared tables and the Snowflake marts are. The two feature tables Databricks builds are its default format in its own Azure storage; making those Iceberg too is a configuration step we have not done.
• Cross-cloud reads cost money and time. It proves the data is not trapped. It is not how you would run a heavy daily workload.
• Storage and catalog were S3 and Glue. The same design applies with Azure Data Lake Storage or Google Cloud Storage and an Iceberg REST, Unity or Polaris catalog; we have not run that yet, and would treat it as its own note.
• Lineage covers what runs through the project. A notebook or a job outside it is invisible until it is brought in.
• Coexistence is a bonus, never a requirement. If you have one engine and are happy, keep it. It may matter if you already run two, are considering a second, or want evidence that moving a workload is technically possible.