Data Governance
Evidence by Construction
Where did this number come from? Who can see it? Is the table open or locked in? Our layer answers all three for every column, on every build — from the dbt project you already have, on Iceberg storage you own, inside your VPC in Australia.
Four things every regulated data team gets asked
Lineage · Quality · Personal data · Ownership — one layer, generated on every build.
Column-level lineage
Click any number, see every source it was computed from.
- —Table and column lineage from the dbt project you already run
- —No migration, no new catalog, no per-seat licence
- —Lives inside your dbt docs: click a table → its documentation; every docs page links back
- —Stored per build in your own database, queryable in plain SQL
Data quality, generated
Tests are written with the pipeline, not after it.
- —Row-count reconciliation between layers
- —Range, set and null expectations derived from the data profile
- —Failures repair themselves before the build goes green
- —Results kept per run for audit
Personal data, followed
A scanner finds email. Lineage finds the six columns derived from it.
- —Classification propagates through every rename, join and hash
- —Masking policy per sensitivity level, ready to apply — hash, redact, partial
- —AI answers respect the same classification
- —Default: classify, don’t block — restrictions only when you decide
Open storage you own
Every table the layer builds is Apache Iceberg on your object storage.
- —Snowflake, Databricks, Athena, ClickHouse read the same files
- —Move engines without moving data
- —Point-in-time snapshots for regulators
- —Nothing leaves your VPC, in Australia
Decouple your data from your engine. Keep one lineage.
Open table formats freed the files from the engine. Governance is still inside it. In our own build, shared tables live once on Apache Iceberg, are produced on one engine and consumed on two others — and columns are still traced end to end across the engine boundary, for everything that runs through the project. Read the 3-minute note →
One stream lands once. A batch engine and a realtime engine read it, independently.
The graph above is a chain: one engine produces, the next two consume what it made, and the lineage crosses each boundary in turn. This one is different. A tick stream is written once as Iceberg by the streaming sink and catalogued once. Snowflake builds the audit-grade marts from it. ClickHouse builds the same models, plus a live quote board, for sub-second reads. Neither reads the other. The same dbt models run on both, and the lineage shows exactly that: two consumers, one copy, no chain.
Measured on 28 September on our own CFD tick stream: four raw tables, 1.9 million ticks each; Snowflake and ClickHouse consumers built and tested; ClickHouse reading the Iceberg files in place through its table function. Snowflake-to-ClickHouse reads are not part of this graph and are not claimed.
| Chain (above) | Batch + realtime (below) | |
|---|---|---|
| Raw layer | Glue-managed Iceberg, daily prices | Glue-catalogued Iceberg, streaming ticks |
| Producer | Athena, a dbt project | Kafka Connect sink, no dbt |
| Consumers | Snowflake, then Databricks reads Snowflake’s mart | Snowflake and ClickHouse, side by side |
| Between consumers | A real dependency, drawn as a boundary | None, drawn as two stacked lanes |
| Same models on two engines | No, each engine has its own models | Yes, six models built on both |
| Measured | 12 tables · 75 edges · 46 cross-engine | 17 tables · 179 edges · 67 cross-engine |
One product with your dbt docs
Same project, same URL, two views. Nothing to install beside dbt docs.
How it works
Point it at your dbt project
Read-only. Nothing changes in your warehouse.
Every build produces the evidence
Lineage, classification, tests — written into dbt docs and your own tables.
Decide what to enforce
Masking, access, human approval — policy in a config table, changed without code.
Made for Australian regulated finance
Runs in your own cloud account, in the region you choose: AWS in our own deployment, with adapters for Azure and Google Cloud. Models you choose. Formats you own. Designed for IRAP-aligned controls.
| Obligation | What you get |
|---|---|
| APRA CPS 230 / CPS 234 | Per-column classification, source-to-report lineage, reproducible per build |
| ASIC reporting | Lineage from source systems to each reported field |
| AUSTRAC record-keeping | Classification and retention rules, snapshot evidence |
| Privacy Act APPs | Personal-information inventory, who can see it, masking evidence |
Open storage used to cost you governance.
Not any more.
Open table formats freed the files from the engine. Lineage, quality evidence and ownership stayed inside each one, so the audit trail stopped at every engine boundary. The layer carries it across, in your own tables.
- —Each engine showed its own lineage; the path stopped at every engine boundary
- —Evidence lived in each tool’s own catalog and screens
- —Lineage of code and lineage of real tables were not distinguished
- —Connecting the engines safely took days of catalog, role and trust configuration
- +One column-level path across every engine and repository
- +Evidence in plain tables in your own database; one SQL view lists every column that crossed an engine
- +Every table marked built or not built, so a picture cannot claim what has not run
- +Repeatable kits that render the exact policies and statements for your environment
See it on your own dbt project
One afternoon, read-only, inside your account. Lineage and a personal-data map you did not have yesterday.