Re-platforming across engines
Add an engine, change an engine or run two side by side without a second copy of the data. One Apache Iceberg copy in your storage, each engine reading it, one column-level lineage that crosses the boundary. Leaving an engine becomes a per-workload decision.
Where you are
Every engine change is a migration: copy the data, rewrite the pipelines, rebuild the lineage, re-prove the controls. So the decision is never made, and the discount at renewal is whatever the vendor offers. Teams that do run two engines run two catalogs and two lineages, and the audit trail stops at the boundary.
How the engagement runs
One copy, open format
Tables land once on Apache Iceberg in your own bucket, catalogued in AWS Glue or an equivalent open catalog. Snowflake, Databricks, Athena and ClickHouse read the same files; none of them owns the data.
Models stay portable
Transformations are dbt, written against dbt's cross-database macros where engines differ. Moving a workload is a change of adapter and profile, not a rewrite.
Lineage that crosses
Column-level lineage is stitched across every dbt repository and engine from their own manifests and compiled SQL, with the cross-engine edges counted. Shared raw tables are matched by their physical identity, so two engines reading one lake show as one source.
Move by workload
Pick the engine per workload on cost, latency or features, keep the evidence of the move in your own tables, and keep the option to move back.
Already built
- ●Four engines reading one Iceberg copy: Snowflake, Databricks, Athena and ClickHouse.
- ●Cross-engine column lineage across four dbt repositories with measured counts, published with the method on the engineering notes.
- ●A real-time estate: Kafka → Iceberg via Kafka Connect with a Glue catalog, consumed independently by Snowflake and by ClickHouse, lineage stitched across both.
BigQuery and Redshift are supported by the pattern and not yet measured on our estate; Azure is mapped layer for layer and stated as planned where it is.
What you bring
- ●An object store (S3, ADLS or GCS) and a catalog, or we configure Glue
- ●The engines you already run, one or several
- ●Existing dbt projects, or SQL we convert into one