DuckDB is one of those projects that has been growing quietly for two or three years and suddenly appears everywhere. What started as the embedded analytical database for analysts with laptops has, as of October 2025, become a piece appearing in small and medium enterprise data architectures with surprising frequency. This article collects the concrete patterns where it is used, where it displaces more expensive pieces, and where it doesn’t reach yet.

Key takeaways

  • DuckDB is an embedded columnar OLAP engine that reads Parquet from local disk or S3 without importing it, treating files as virtual tables.

  • The most common enterprise pattern is replacing a small cloud warehouse (100-500 GB, hundreds of queries per day) with DuckDB plus Parquet in object storage.

  • Aggregation queries SQLite takes twenty seconds to run, DuckDB resolves in half a second; the difference is the columnar storage model.

  • Limits are clear: no write concurrency, no horizontal scale, and no native governance layer.

  • If your analytical load has less than 500 GB of active data and fewer than 10 queries per second, DuckDB deserves serious evaluation before assuming you need a distributed warehouse.

What makes it different

DuckDB is a columnar OLAP database that runs embedded in the process using it. As the project’s own documentation[1] puts it, there is "no DBMS server software to install, update and maintain": no deployment, no resource consumption when not queried. Import it as a library, open a database on disk or in memory, and run standard SQL with its catalog of analytical extensions.

The key difference with SQLite is the storage engine, and we covered the comparison in more depth in SQLite and DuckDB: when each is the right choice. SQLite is row-based and optimized for small transactions with high write concurrency. DuckDB is columnar and optimized for analytical scans over millions of rows.

An aggregation query SQLite takes twenty seconds to run, DuckDB runs in half a second. Not magic: DuckDB’s vectorized engine processes columns in batches, and that storage model is built for exactly what analytics does.

The second important difference is native Parquet support. DuckDB can read Parquet files from local disk or S3 without importing them first, treating them as virtual tables. This changes the architecture equation: Parquet as primary storage format, DuckDB as query engine, no warehouse in between.

Replacing small warehouses

The first recurring pattern is replacing a small cloud warehouse with DuckDB plus Parquet in object storage. In the cases accompanied, these were architectures where Snowflake or BigQuery were used with small data volumes, between 100 and 500 gigabytes. Query patterns were punctual, a few hundred queries per day, and monthly costs did not match real usage.

The typical replacement:

  • Write data as partitioned Parquet in an object storage bucket.

  • Use DuckDB in a server or function to run queries.

Storage cost drops to cents per gigabyte per month, and compute cost is limited to real query minutes. In one project, the monthly bill dropped from two thousand dollars to under one hundred, without losing real capability.

Analytics engine behind APIs

The second pattern is using DuckDB as the query engine behind data APIs. DuckDB enables an alternative to expensive OLAP replication: generate periodic extracts in Parquet, query them directly with DuckDB from the API service, and serve analytics responses with millisecond latency.

The pattern fits especially well for internal APIs or enterprise dashboards where data doesn’t have to be minute-fresh. An update every fifteen minutes or every hour is enough for most executive dashboards, and it simplifies the architecture a lot. Teams coming from Redshift or dedicated ClickHouse for this case have dropped operational complexity to zero while keeping equivalent or better response times.

One important detail: DuckDB is not built for high write concurrency. If more than one process tries to write to the same DuckDB file at once, you get locking. The architecture that works is to keep writes and reads separate.

Processing large files with no infrastructure

The third pattern is the most surprising because of its simplicity. DuckDB can process CSV, JSON, or Parquet files of hundreds of gigabytes from the command line with no infrastructure at all. An analyst with a laptop and DuckDB CLI can run queries on files that previously required a Spark cluster or an ETL job in the warehouse.

The internal mechanism is that DuckDB uses temporary disk spill when memory doesn’t suffice, with an execution algorithm that respects configured limits. A JOIN query between two 50-gigabyte files runs, perhaps slowly, but it runs, on a laptop with 16 gigabytes of RAM.

This pattern surprises most in analysis labs, in one-off data-reconciliation tasks, and in audits. A team that needs to cross-check a bank file against an ERP extract can do it with a single DuckDB command and get results in minutes, without importing anything into a dedicated tool.

How it fits with dbt and Python

DuckDB has integrated well with two ecosystems that particularly matter in enterprise: dbt and Python.

For dbt there is the dbt-duckdb adapter, documented on dbt’s own site[2], which lets teams treat DuckDB like any other engine. Pipelines can run locally with no connection to the production warehouse, and the simplest configuration doesn’t even write to disk: it runs in memory. That capability covers development and testing, keeping Snowflake or BigQuery only for production. This connects directly to the stack we covered in modern data engineering with Iceberg and dbt.

For Python, DuckDB integrates with pandas, Polars, and Arrow smoothly, something we covered in more depth in Polars versus pandas in 2025: the real-world practice. You can pass a pandas DataFrame as a table to a DuckDB query and get the result back as a DataFrame, with no copy involved. The Polars plus DuckDB combination is particularly interesting. Both are high-performance analytical engines, and going from one to the other via the Arrow format is zero-copy, as the Apache Arrow project itself documents[3].

Real limits

Not everything is positive. Three areas where DuckDB remains weak in 2025:

  • Write concurrency: not designed for more than one process writing simultaneously. For continuous ingestion from more than one producer, additional patterns or alternatives like ClickHouse are needed.

  • Horizontal scale: runs in a single process, on a single machine. The practical boundary is around one or two terabytes of actively queried data.

  • Governance: no native permissions, auditing, or access-control layer. In sectors where regulatory compliance carries real weight, these capabilities must be built around it.

My read

DuckDB has earned its spot in enterprise analytics. It doesn’t replace Snowflake or BigQuery in cases where scale justifies them. It does replace underused warehouses, OLTP databases misused for analytics, Spark clusters oversized for small loads, and ClickHouse installations set up for volumes that didn’t need it. For all of those cases, DuckDB lowers cost, lowers operational complexity, and raises speed.

The concrete recommendation: if your analytics load has less than 500 gigabytes of active data and fewer than 10 queries per second, DuckDB deserves serious evaluation before assuming you need a distributed warehouse. The proof of concept takes days, and the outcome that keeps repeating is ten times better on cost.

Conclusion

DuckDB does one thing well, and that also means there are things it doesn’t do. Recognizing where it fits and where it doesn’t is part of engineering, not a project defect.

This article is also available in Spanish: DuckDB en analítica de empresa: casos concretos.

Sources:

  1. DuckDB: Why DuckDB, embedded architecture and columnar engine[1]
  2. dbt Labs: DuckDB setup, the dbt-duckdb adapter[2]
  3. Apache Arrow: DuckDB quacks Arrow, a zero-copy data integration[3]

Frequently asked questions

Up to what data volume can I replace Snowflake or BigQuery with DuckDB?

If your load has less than 500 GB of active data and fewer than 10 queries per second, DuckDB deserves serious evaluation before assuming you need a distributed warehouse. The pattern is to write partitioned Parquet into an object storage bucket and query it with DuckDB from a server or function. In one project the monthly bill dropped from two thousand dollars to under one hundred. With terabytes or dozens of concurrent users, warehouses make sense again.

Can DuckDB process a file larger than my laptop's RAM?

Yes. DuckDB spills temporarily to disk when memory doesn't suffice, with an execution algorithm that respects configured limits. A JOIN between two 50 GB files runs, perhaps slowly, on a laptop with 16 GB of RAM. From the CLI it can process CSV, JSON or Parquet files of hundreds of gigabytes with no infrastructure at all, which suits one-off data reconciliation tasks and audits.

What can't DuckDB do compared with an enterprise warehouse?

Three things. It is not built for write concurrency: more than one process writing to the same file causes locking, so keep writes and reads separate or use alternatives like ClickHouse for continuous ingestion. It doesn't scale horizontally: it runs in a single process on a single machine, with a practical boundary around one or two terabytes of actively queried data. And it has no native permissions, auditing or access-control layer, which must be built around it.

Sources

  1. the project’s own documentation
  2. the dbt-duckdb adapter, documented on dbt’s own site
  3. the Apache Arrow project itself documents