Updated: 2026-07-07

DuckDB is one of those projects that has been growing quietly for two or three years and suddenly appears everywhere. What started as the embedded analytical database for analysts with laptops has, as of October 2025, become a piece appearing in small and medium enterprise data architectures with surprising frequency. This article collects the concrete patterns where it is used, where it displaces more expensive pieces, and where it doesn’t reach yet.

Key takeaways

  • DuckDB is an embedded columnar OLAP engine that reads Parquet from local disk or S3 without importing it, treating files as virtual tables.

  • The most common enterprise pattern is replacing a small cloud warehouse (100-500 GB, hundreds of queries per day) with DuckDB plus Parquet in object storage.

  • Aggregation queries SQLite takes twenty seconds to run, DuckDB resolves in half a second; the difference is the columnar storage model.

  • Limits are clear: no write concurrency, no horizontal scale, and no native governance layer.

  • If your analytical load has less than 500 GB of active data and fewer than 10 queries per second, DuckDB deserves serious evaluation before assuming you need a distributed warehouse.

What makes it different

DuckDB is a columnar OLAP database that runs embedded in the process using it. As the project’s own documentation[1] puts it, there is "no DBMS server software to install, update and maintain": no deployment, no resource consumption when not queried. Import it as a library, open a database on disk or in memory, and run standard SQL with many analytical extensions.

The key difference with SQLite is the storage engine, and we covered the comparison in more depth in SQLite and DuckDB: when each is the right choice. SQLite is row-based and optimized for small transactions with many concurrent writes. DuckDB is columnar and optimized for analytical scans over millions of rows. An aggregation query SQLite takes twenty seconds to run, DuckDB runs in half a second. Not magic: DuckDB’s vectorized engine processes columns in batches, and that storage model is built for exactly what analytics does.

The second important difference is native Parquet support. DuckDB can read Parquet files from local disk or S3 without importing them first, treating them as virtual tables. This changes the equation of many architectures: Parquet as primary storage format, DuckDB as query engine, no warehouse in between.

Replacing small warehouses

The first pattern appearing often is replacing a small cloud warehouse with DuckDB plus Parquet in object storage. In the cases accompanied, these were architectures where Snowflake or BigQuery were used with small data volumes (between 100 and 500 gigabytes), punctual query patterns (a few hundred queries per day), and monthly costs not matching real usage.

The typical replacement:

  • Write data as partitioned Parquet in an object storage bucket.

  • Use DuckDB in a server or function to run queries.

Storage cost drops to cents per gigabyte per month, and compute cost is limited to real query minutes. In one project, the monthly bill dropped from two thousand dollars to under one hundred, without losing real capability.

Analytics engine behind APIs

The second pattern is using DuckDB as the query engine behind data APIs. DuckDB enables an alternative to expensive OLAP replication: generate periodic extracts in Parquet, query them directly with DuckDB from the API service, and serve analytics responses with millisecond latency.

The pattern fits especially well for internal APIs or enterprise dashboards where data doesn’t have to be minute-fresh. An update every fifteen minutes or every hour is enough for most executive dashboards, and it simplifies the architecture a lot. Teams coming from Redshift or dedicated ClickHouse for this case have dropped operational complexity to zero while keeping equivalent or better response times.

One important detail: DuckDB is not built for high write concurrency. If several processes try to write to the same DuckDB file at once, you get locking. The architecture that works is to keep writes and reads separate.

Processing large files with no infrastructure

The third pattern is the most surprising because of its simplicity. DuckDB can process CSV, JSON, or Parquet files of hundreds of gigabytes from the command line with no infrastructure at all. An analyst with a laptop and DuckDB CLI can run queries on files that previously required a Spark cluster or an ETL job in the warehouse.

The internal mechanism is that DuckDB uses temporary disk spill when memory doesn’t suffice, with an execution algorithm that respects configured limits. A JOIN query between two 50-gigabyte files runs, perhaps slowly, but it runs, on a laptop with 16 gigabytes of RAM.

This pattern surprises most in analysis labs, in one-off data-reconciliation tasks, and in audits. A team that needs to cross-check a bank file against an ERP extract can do it with a single DuckDB command and get results in minutes, without importing anything into a dedicated tool.

How it fits with dbt and Python

DuckDB has integrated well with two ecosystems that particularly matter in enterprise: dbt and Python.

For dbt there is the dbt-duckdb adapter, documented on dbt’s own site[2], which lets teams treat DuckDB like any other engine, with the benefit that pipelines can run locally with no connection to the production warehouse (the simplest configuration doesn’t even write to disk, it runs in memory). Many teams use this for development and testing, keeping Snowflake or BigQuery only for production. This connects directly to the stack we covered in modern data engineering with Iceberg and dbt.

For Python, DuckDB integrates with pandas, Polars, and Arrow very smoothly, something we covered in more depth in Polars versus pandas in 2025: the real-world practice. You can pass a pandas DataFrame as a table to a DuckDB query and get the result back as a DataFrame, with no copy involved. The Polars plus DuckDB combination is particularly interesting: both are high-performance analytical engines, and going from one to the other via the Arrow format is zero-copy, as the Apache Arrow project itself documents[3].

Real limits

Not everything is positive. Three areas where DuckDB remains weak in 2025:

  • Write concurrency: not designed for many processes writing simultaneously. For continuous ingestion from several producers, additional patterns or alternatives like ClickHouse are needed.

  • Horizontal scale: runs in a single process, on a single machine. The practical boundary is around one or two terabytes of frequently-queried data.

  • Governance: no native permissions, auditing, or access-control layer. In sectors where regulatory compliance carries real weight, these capabilities must be built around it.

My read

DuckDB has earned its spot in enterprise analytics. It doesn’t replace Snowflake or BigQuery in cases where scale justifies them, but it replaces underused warehouses, OLTP databases misused for analytics, Spark clusters oversized for small loads, and ClickHouse installations set up for volumes that didn’t need it. For all of those cases, DuckDB lowers cost, lowers operational complexity, and raises speed.

The concrete recommendation: if your analytics load has less than 500 gigabytes of active data and fewer than 10 queries per second, DuckDB deserves serious evaluation before assuming you need a distributed warehouse. The proof of concept usually takes days, and the result, more often than not, is an order of magnitude better on cost.

Conclusion

DuckDB does one thing very well, and that also means there are things it doesn’t do. Recognizing where it fits and where it doesn’t is part of engineering, not a project defect.

This article is also available in Spanish: DuckDB en analítica de empresa: casos concretos.

Sources

  1. the project’s own documentation
  2. the dbt-duckdb adapter, documented on dbt’s own site
  3. the Apache Arrow project itself documents