ACBDiDATAP.AI
← datap.ai home

Reference Data & AI Layer Architecture

One layer between your people and your data platform. Iceberg at the bottom, governance carried up to every column, AI on top — inside your VPC.

AWS-nativeCustomer-VPC deployableApache IcebergAFSL · APRA · NIST · ISO 42001Zero data migration
Where the layer sits

Your people. Our layer. Your engines. Your storage.

Four layers, top to bottom. Layers 1, 3 and 4 are yours and stay as they are. Layer 2 is what we add — inside your VPC, in your region.

1YOUR PEOPLE & APPSUnchanged. They just get better data — and answers they can trust.BI dashboardsAnalysts · SQLAI assistantsRegulator reportsApps & APIsYour domain apps2DATAP.AI · DATA & AI LAYERA layer, not a replacement. Runs in your VPC, in your region.Own your dataApache Iceberg · any enginePipelines that build themselvesplanned · generated · self-repairedGovernance to the columnlineage · personal data · quality · AI guardrailsSpeak every market8 APAC languagesMODELS YOU CHOOSE — LOCAL OR IN-REGIONClaudeGPTGeminiLlama · localQwen · localBedrock · Vertex · Azure3YOUR COMPUTE ENGINESUnchanged. Add one, drop one — the layer works across all of them.SnowflakewarehouseDatabrickslakehouseBigQuerywarehouseRedshiftwarehouseAthena · GlueserverlessClickHousereal-timeTrino · Sparkopen engines4OPEN STORAGE · APACHE ICEBERGOne format you own. Every engine above reads the same files — no copies, no lock-in.AWS S3Sydney · Singapore · TokyoAzure Data Lake StorageAustralia East · Southeast AsiaGoogle Cloud StorageSydney · Singapore · TokyoOpen-source foundations: Apache Iceberg · dbt · Airflow · LangChain · LangGraph · AutoGen · Lightdash

Four design principles

01
Kafka is the spine

Every event passes through Kafka. ClickHouse (realtime) and S3 Iceberg (lake) are parallel sinks, optimised for different time horizons. ClickHouse is not OLTP — transactional state lives in Aurora PostgreSQL.

02
Iceberg unifies the lake

Snowflake writes Iceberg-format tables. ClickHouse reads the same physical files via Snowflake Open Catalog (REST) or AWS Glue — one set of parquet files, multiple engine readers, no copy.

03
Governance is wired in

dim_ai_control rules pre/post-screen every AI Trade Council output for AFSL (s761G + 912A), MAS Notice FSG-N16, NIST AI RMF, ISO 42001. Audit trail in S3 Object Lock + Iceberg is auditor-ready for ASIC inspection.

04
BI is open, semantic-layer-first

dbt is the transformation + governance layer; Lightdash auto-binds to dbt models. The semantic layer survives BI tool swaps. Zero per-seat license cost.

Looking for the ingestion-to-serving stack and the production runtime view? They live on Data & AI Pipelines →