Modern data engineering: dbt, Iceberg and the lakehouse come together
Updated: 2026-07-07
The analytical data stack has spent the last decade promising a clean separation between storage, engine and transformation, but only in the last twelve months has that promise stopped sounding like a brochure. Apache Iceberg with REST catalogs, dbt as a declarative layer, and an interchangeable query engine underneath have become the real skeleton of new analytics projects. I am writing this down because the pattern is repeating across too many teams to ignore, and because the decisions made now shape the next five years of a data platform.
Key takeaways
-
Iceberg has won the open table format adoption war for mundane rather than technical reasons: Databricks, Snowflake, BigQuery and Redshift all support it natively.
-
Iceberg’s REST catalog, part of the spec since 2022, became the real glue between engines in 2025: per the latest community survey, 60.7% of teams use Trino and 28.6% use DuckDB over the same tables.
-
dbt Fusion, the Rust rewrite of dbt-core, introduces static column and type validation at compile time; Iceberg support is still under active development adapter by adapter.
-
Interactive query performance over Iceberg in an open engine still trails a well-tuned closed warehouse, sometimes by a factor of two or three.
-
The open lakehouse with Iceberg and dbt pays off when three conditions hold simultaneously: sufficient volume, workload heterogeneity, and data engineering maturity.
Why Iceberg won the table-format war
When the open table format conversation started, the three candidates were Iceberg, Delta Lake and Hudi. By late 2025 the reality is that Iceberg has won the adoption war for mundane rather than technical reasons:
-
Databricks paid over a billion dollars for Tabular[1] in June 2024, the company founded by Iceberg’s original creators, and has been working to converge Delta and Iceberg ever since.
-
Snowflake has supported Iceberg natively for reads for a while and reached general availability for writes to externally managed Iceberg tables in October 2025[2]; BigQuery and Redshift also read it natively.
-
AWS launched Amazon S3 Tables[3] in December 2024, a managed bucket type that uses Iceberg as its native format with automatic compaction and maintenance.
When your three main warehouses and the largest open analytics engine converge on the same format, the choice stops being technical.
The interesting part is that Iceberg isn’t best at any single aspect individually. Delta has a tighter, faster engine inside Databricks; Hudi remains superior for frequent streaming updates. But Iceberg is the one that kept the cleanest abstraction between specification and engine: its metadata is JSON and Avro, its hidden partitioning model avoids classic ill-written-predicate pitfalls, and its time-travel and schema-evolution support is the most rigorous of the three.
The 2025 twist hasn’t been technical, it has been adoption. The REST catalog specification[4] has existed since the project’s 0.14 release in 2022, but only this year has it filled up with serious open implementations (Apache Polaris, Lakekeeper, an open Unity Catalog) competing to be the single service that talks to every engine. Today you can have Trino, Spark, DuckDB and a Python client all reading and writing the same tables through one catalog, with none of them getting along better with it than the rest. For anyone coming from Delta Lake, the underlying comparison in Delta Lake versus Iceberg is still the mandatory starting point before picking a format.
dbt on top: what Fusion changes
The transformation layer has evolved in parallel. dbt Labs unveiled the Fusion engine in May 2025[5], a Rust rewrite of dbt-core that compiles the model graph, validates SQL before executing, and emits optimized execution plans, with up to 30x faster parsing and twice the speed on full-project compilation. The old Python-based engine keeps working for existing projects, but Fusion introduces static column and type validation at compile time, something long missing that has caused many wasted hours chasing errors an analyzer should have caught earlier.
What matters isn’t just the speed. What matters is where the project is headed: first-class Iceberg support is under active development inside Fusion, adapter by adapter, with DuckDB and Spark already in public beta. For teams that until now were locked into pure Snowflake or BigQuery, this opens the door to decoupling the storage platform from the query engine without rewriting models, though in 2026 you still have to check case by case which adapter already supports Iceberg and which doesn’t yet.
The combination of dbt plus Iceberg solves the real problem most midsize data platforms face: vendor lock-in doesn’t come from the engine, it comes from the data. If your tables live in Iceberg with a REST catalog, moving from Snowflake to Trino, or trying DuckDB on top of the same lake, stops being a full rewrite and becomes a connection change. This complements the patterns covered in DuckDB in enterprise analytics: the same embedded engine that serves local analytics can reach the same Iceberg tables in the lake.
Where it still hurts
Not everything is solved. Interactive query performance over Iceberg in an open engine like Trino still trails a well-tuned closed warehouse, sometimes by a factor of two or three. The reason is that Snowflake and BigQuery have query optimizers, result caches and resource management refined over a decade that open engines can’t replicate with four engineers. For operational dashboards running dozens of queries per second, open Iceberg is still too cold.
The second issue is operating the catalog itself. A REST catalog is critical infrastructure: if it goes down, every engine loses a coherent view of the tables. Commercial vendors like Tabular handle this, but self-hosting it yourself with Lakekeeper means high availability, metadata backups, and a recovery plan that most teams never needed while they lived inside Snowflake and the catalog was part of the service.
The third front is write cost. Iceberg optimizes for analytical reads with many partitions and large Parquet files, but keeping that optimum requires regular compaction and cleanup jobs. Forgetting to compact a streaming-ingested table leads within weeks to thousands of small files and minute-long queries. Closed warehouses do this for you in the background; with Iceberg you have to orchestrate it by hand.
When it pays off
The open lakehouse with Iceberg and dbt pays off when three conditions hold simultaneously:
-
Enough volume for the closed warehouse cost to be a significant line item in the budget, typically above six figures annually. Below that, the savings don’t cover the operational cost of running the catalog and the compaction jobs.
-
Workload heterogeneity. If all your analysis is interactive SQL over tabular data, a closed warehouse gives you a better experience. Iceberg shines when you need to mix Python, Spark, interactive SQL and machine learning models over the same tables without duplicating data.
-
Data engineering maturity to operate your own infrastructure. Iceberg isn’t Snowflake: it doesn’t get configured in a web panel and just work. Teams arriving from a closed warehouse often underestimate this curve and end up with a worse-run lakehouse than the warehouse they left behind.
When in doubt, I would start with a small pilot on a non-critical dataset before committing the entire platform.
Conclusion
The Iceberg plus dbt stack is today the most sensible answer for organizations wanting to escape vendor lock-in without giving up quality declarative transformations. The tools have matured enough for the operational friction to be reasonable, but the learning curve is real and shouldn’t be underestimated.
