https://platform.datap.ai DATAP.AIplatform.datap.ai · one console FOUR INTELLIGENCES AI › Text-to-SQL AI › AI Agent Hub BI · CI · DI (Airflow, dbt, Airbyte) GOVERNANCE Cross-engine lineage Column lineage (dbt) AI governance gate HITL approvals QUERY ENGINES Athena (AWS) Snowflake Databricks (Azure) BigQuery · Redshift · ClickHouse CATALOGS Glue Catalog (AWS) Unity Catalog (Azure) DATA STORAGE S3 · Iceberg (AWS) ADLS · Iceberg (Azure) DATAP.AI · every scene has run in 2026 · Databricks (Azure) · dbt-core · Spark SQL · Iceberg · GenAI · agentic AI 1 · The platform: open storage, one catalog, Databricks on top Data stays the customer's asset in their storage. Databricks is the engine; the same dbt models also ran on Athena and Snowflake. STORAGE +CATALOG Apache Iceberg tables in the customer's own storage (S3 today; ADLS Gen2 in the design) Catalogs: Glue (AWS) · Unity Catalog (Azure). One physical table, seen by three catalogs, no copy per engine. TRANSFORMDatabricks · dbtSpark SQL dbt-core on Azure Databricks · SQL warehouse · Unity Catalog chinook_staging (11 views) → chinook_core (dim_customer, dim_track, fct_invoice …) → chinook_marts dbt build · PASS 92 / 92 · docs generated · Snowflake dialect rewritten as Spark SQL (explode(sequence(...))) Second Databricks project: ML feature marts of a stock estate (8.1M rows, 84 tickers) reading the same Iceberg copy that the Athena producer writes. Three repositories, one estate file, one lineage. github.com/codep-ai: chinook-dbt-databricks · xengine-producer-stock · xengine-consumer-analytics · -ml SERVE Airflow (38 DAGs) · Lightdash BI on the Databricks warehouse · Text-to-SQL · column lineage All from this console. Engines, catalogs and storage are menu entries, not the architecture. Azure Databricks · Unity Catalog dbt-core · 22 models Spark SQL via dbt-databricks Iceberg · open format Also on Athena + Snowflake Why this shape: the engine is a choice per team; the data, the models and the lineage are the shared asset. Storage → transform → serve, read bottom-up. Every box below has run in the estate; the counts are from the 22–24 Sep builds. 2 · dbt column lineage across engines: Snowflake → Databricks dbt docs stops at each project's boundary. This stitches three dbt manifests by physical table identity, so lineage crosses engines and repos. Athena (AWS) · dbt producerrepo xengine-producer-stock Snowflake · dbt consumerrepo xengine-consumer-analytics Databricks (Azure) · dbt consumerrepo xengine-consumer-ml model · common_price_daily tickerexchangetrade_datecurrency_codecloseunits_per_usd source · Raw Iceberg (Glue-managed)stock_iceberg.prices · 8.5M rowsexchange_ref · fx_rates_dailyticker_universe model · mart_price_monthly_usd tickerexchangemonth_startcurrency_codecompany_name · sectoravg_close_usd reads the Iceberg tables in placeGlue-synced external tablesreader grants · no copy391,271 mart rows model · feat_exchange_monthly exchangecurrency_codemonth_starttickersavg_close_usdtotal_volume model · feat_price_momentumticker · exchange · sectortrade_date · close_usd8,085,051 rows · Unity Catalog Measured 22 Sep: 12 tables · 75 columns · 46 cross-engine column edges · 0 unresolved · same result in one repo or three. Built: a stitcher over dbt manifests + catalogs, evidence tables per build, and a "built" flag so a stale run cannot pose as lineage. Each engine governs what lives inside it. This governs what crosses between them. Live, auto-walking graph of this run: datap.ai/data-governance 3 · GenAI on top: Text-to-SQL that speaks Spark SQL to Databricks RAG over DDL, documentation and past questions; dialect-aware prompt; executed on the Databricks SQL warehouse. Real answer, 24 Sep. AI · Text-to-SQL · Database: Databricks (Azure) · Run on Databricks SQL warehouse (direct) Question
Which artists have the most albums? Top 5.
 Generated Spark SQL SELECT ar.name AS artist, COUNT(al.album_id) AS albums FROM azure_datapai_databricks_dev.chinook.album al JOIN azure_datapai_databricks_dev.chinook.artist ar ON al.artist_id = ar.artist_id GROUP BY ar.name ORDER BY albums DESC LIMIT 5 Result · Databricks SQL warehouse artistalbums Iron Maiden21 Led Zeppelin14 Deep Purple11 U210 Metallica10 Same RAG store, three dialects Snowflake · Athena/Trino · Spark SQL (Databricks) Built: engine-aware prompt + validation (three-part Unity names), schema context from Unity Catalog, neutral LLM client. Guardrails: read-only statements, row caps, cost guard, every call logged. The model provider is a config value, not a dependency. Snowflake path also runs through Lightdash (SQL Runner + save to chart); the Databricks warehouse is now registered there. 4 · Agentic AI on top: agents build the pipeline, governance gates it AG2 (AutoGen) agents with audited tools: profile the source, check keys, scan PII, generate staging SQL + YAML, save dbt artifacts. A person approves. orchestratorplans · routes · stopsConversableAgent ingest_agentingest_file · profile_table quality_agentcheck_primary_key compliance_agentscan_pii · mask_pii_columns transform_agentgenerate_staging_sql · yaml Output: dbt artifacts, not a black box models/staging/stg_<source>.sql models/staging/_stg_<source>.yml sources.yml · audit log per call Reviewed in git, built by dbt, lineage inherits. GOVERNANCEGATE HITL approvals · AI governance controls mapped to frameworks · cost guard · provider policy (which model sees which data) Agents propose; a person approves; the platform records who, what and when. The same rules the Text-to-SQL path obeys. Provider-neutral LLM client: AWS Bedrock (with Guardrails), Anthropic, OpenAI, or a local Ollama model, chosen by config for residency. ag2_pipeline/{pipeline, tools, audit, compliance_tools, cost_control, llm_config}.py The order matters: platform, then governance that travels, then GenAI, then agents. Each layer is only as trustworthy as the one below it. datap.ai · platform.datap.ai · github.com/codep-ai · happy to walk through any layer live