Your data is your asset.
Not your engine vendor’s.
Technology leaders tell us the same thing: the data must stay the company’s own asset, not be bound to whichever engine processes it today. Three things make that true: an open format in storage you own, compute you can swap, and governance that travels with the data instead of living inside an engine. This page walks through all three, on a build we ran. It applies on AWS, Azure or Google Cloud; the layer lives in your account, not ours.
The same table, seen from three catalogs
Not a diagram: the real definitions of one shared table in our build, as each catalog reports it. Every engine points at the same metadata file in the same bucket. Nobody holds a copy.
Name common_price_daily
Database stock_common_mesh
TableType EXTERNAL_TABLE
table_type ICEBERG
metadata_location s3://codepais3/stock/bronze/_xengine_mesh/
stock_common_mesh/common_price_daily/
<table-uuid>/metadata/00000-….metadata.json
Columns ticker string, exchange string,
trade_date date, currency_code string,
close double, units_per_usd double,
close_usd double, volume doubleSHOW ICEBERG TABLES … STOCK_COMMON_MESH
name COMMON_PRICE_DAILY
external_volume_name DATAPAI_S3_VOL
catalog_name GLUE_STOCK_COMMON
catalog_table_name common_price_daily
iceberg_table_type UNMANAGED -- Snowflake does not own it
SYSTEM$GET_ICEBERG_TABLE_INFORMATION(…)
metadataLocation s3://codepais3/stock/bronze/_xengine_mesh/
stock_common_mesh/common_price_daily/
<table-uuid>/metadata/00000-….metadata.json
status successDESCRIBE EXTENDED snowflake_datapai_iceberg
.stock_common_mesh.common_price_daily
Type FOREIGN
Provider snowflake -- Glue-owned: served through Snowflake
columns TICKER, EXCHANGE, TRADE_DATE, CURRENCY_CODE,
CLOSE, UNITS_PER_USD, CLOSE_USD, VOLUME
DESCRIBE EXTENDED snowflake_datapai_iceberg
.stock_consumer_mesh.mart_price_monthly_usd
Type EXTERNAL
Provider iceberg -- Snowflake-managed: read from S3 directly
Metadata s3://codepais3/stock/bronze/_dbt/
STOCK_CONSUMER_MESH/mart_price_monthly_usd/
<table-uuid>/metadata/00001-….metadata.jsonCaptured 22 September 2026 from our own accounts. The Glue-owned table reaches Databricks in Azure through Snowflake, because a Glue catalog was not among the documented federation sources for Azure Databricks when we built this; the Snowflake-managed mart is read by Databricks straight from the S3 metadata file. Both routes end at files in the customer’s bucket.
One copy, on Apache Iceberg, in a bucket you own
Sovereignty starts with where the bytes live and who can read them. The shared tables exist once, as Apache Iceberg files in object storage you own — Amazon S3, Azure Data Lake Storage or Google Cloud Storage; Iceberg is the same format on all three — registered in an open catalog. No engine owns them.
Open table format
Iceberg is an open specification with readers in every major engine. Schema evolution, snapshots and time travel are properties of the files, not of any vendor.
Your cloud, your account, your region
S3, Azure Data Lake Storage or Google Cloud Storage: the files sit in your account, in the region you choose. Engines are given read access to them; they never receive a copy. In our build the storage was S3; the design does not depend on that.
An open catalog of record
The producer registers its tables in a catalog that engines other than its own can read. In our build that was AWS Glue; an Iceberg REST catalog, Unity Catalog or Polaris can play the same role. Snowflake followed it through a catalog integration; Databricks reached the same files through a foreign catalog. One registration, many readers.
Written to be shared
Shared tables are merge models, so a table keeps one identity across runs and every consumer keeps following it. A rebuilt table gets a new identity, and consumers silently stop; we learned that the hard way.
If you left every engine tomorrow, the files, their history and the catalog entries would still be yours, readable by whatever you choose next.
Athena produces. Snowflake and Databricks consume. Same models, two clouds.
Once the data is in an open format in your storage, the engine becomes a choice per workload instead of a decision for a decade. In our build one team produces on Athena; an analytics team consumes on Snowflake; an ML team consumes on Databricks in Azure.
The same model code everywhere
dbt-core with one adapter per engine. The models are the same SQL; only the target changes. First as one project, then, to show the enterprise shape, as a repository per team.
Small engine, big engine, by choice
Reference tables and daily merges run on a serverless engine at low cost. Heavy marts run where the warehouse is. Feature tables run where the ML platform is. Each placement is revisable.
Across clouds, without copying
The Databricks consumer in Azure reads the Snowflake-managed Iceberg tables straight from S3 and the Athena-produced tables through Snowflake. No table data moved between engines; row counts and values matched on every engine.
What a second engine really costs
Not a migration: an adapter in the environment, a profile, and one catalog bridge so it can find the tables. The kits that render the exact statements are part of the layer.
You can add an engine for one workload, or leave one, without touching the data or the other engines. That is what replaceable means.
One column-level lineage across every engine, held in your own tables
Open files solved half of the problem. Lineage, quality evidence and ownership still lived inside each engine, so the audit trail stopped at every engine boundary. The layer carries it across.
Lineage stitched across engines
Each engine’s graph ends at a source; the layer joins the graphs where a consumer’s source and a producer’s model declare the same physical table. In our build: 12 tables, 75 column edges, 46 crossing an engine boundary, 0 unresolved.
Evidence as tables, not screens
Every edge is written to your database with the engine on both sides; every table with whether it was actually built. An auditor answers “where did this number come from” with SQL.
Built, not just compiled
A table is marked built only if the engine’s own catalog lists it with columns. A picture cannot claim what has not run.
Across repositories as well as engines
The same lineage held when the models were split into a repository per team. The join is on physical identity, not on project membership.
Switch the layer off and the evidence tables and their history remain. Governance that lives in your database is governance you own.
Your estate is on Azure? Same design.
Most banks and government agencies we talk to run on Azure. Nothing above depends on AWS: Iceberg is the same format on Azure Data Lake Storage, the catalog and the engines have Azure equivalents, and the governance layer never reads a bucket. Here is the mapping, with what our published build has and has not exercised on Azure.
| Layer | On Azure | Status in our build |
|---|---|---|
| Object storage you own | Azure Data Lake Storage Gen2 (the same Apache Iceberg files; the format does not change) | Planned build: our published storage was S3 |
| Open catalog of record | Unity Catalog’s Iceberg REST endpoint, or Snowflake Open Catalog (Polaris) | Planned: Glue played this role in our build |
| Producing engine | Azure Databricks (serverless SQL or a small cluster), or Microsoft Fabric | Planned: Athena produced in our build |
| Consuming engines | Snowflake on Azure; Azure Databricks; open engines such as Trino or DuckDB over the same files | Azure Databricks consumer: already real in our build, reading Iceberg from another cloud |
| Governance layer | Same lineage stitcher, estate file, evidence tables; identities gain Azure forms | Cloud-agnostic today; nothing in it reads S3 |
| Models | Azure OpenAI with managed identity, or local models on your own hosts | Wired today; see AI Sovereignty |
See it: one lineage across three engines and three repositories
Live, from the three-repository run. Click any column; its provenance lights up in both directions across every engine and repository it passes through.
The evidence, in three notes
We built it before we wrote about it. Each note says what ran, what we measured, what broke and what it does not show.
Your data is your asset. Not your engine vendor’s.
The pain, where standard tooling stopped in our own build, what the layer adds, and the business value.
Three engines, one shared Iceberg copy, one lineage
Architecture, catalog routing, the code we had to build, what we measured, five things that broke, strategy and roadmap.
Three teams, three repositories, one lineage
The same models split into a repository per team. Identical tables, identical lineage, and the measured cost of adding a team.
Read the code
The team repositories of the multi-team build are public. The claims about what a new team touches, and what it does not, can be checked there.
The same independence, one layer up: AI sovereignty.
Models you choose, running where you choose, answering only from data whose lineage and classification you hold.
Everything on this page and in the notes was built and measured on our own demonstration environment with sample data. It is not a production reference and not a benchmark. The build used S3 and Glue for storage and catalog, with Snowflake and Databricks on Azure as consumers; we have not yet run the storage side on Azure Data Lake Storage or Google Cloud Storage. Questions of legal jurisdiction, encryption-key control and who operates the platform are separate, and the demonstration does not address them. Apache Iceberg is a trademark of The Apache Software Foundation. dbt is a trademark of dbt Labs. Amazon Athena, AWS Glue and Amazon S3 are trademarks of Amazon.com, Inc. or its affiliates. Snowflake is a trademark of Snowflake Inc. Databricks and Unity Catalog are trademarks of Databricks, Inc. Microsoft Azure is a trademark of Microsoft. DATAP.AI is not affiliated with, endorsed by or sponsored by any of them.