Data engineer swaps complex big data stacks for DuckDB and DuckLake

After nearly two decades in data engineering, Matt Martin has moved away from heavy distributed computing frameworks in favor of simpler, SQL-based tools. In a recent case study published by the DuckDB team on October 29, 2026, Martin explains how he replaced complex infrastructure with DuckDB and its new table format, DuckLake. His journey highlights a growing trend among engineers who prioritize ease of use and lower maintenance over raw scale that their workloads do not actually require.

What happened

Martin’s career began with manual spreadsheet manipulation and VBA macros before evolving into automated SQL pipelines. As data volumes grew, he adopted industry-standard big data tools like Teradata, Google BigQuery, and Apache Spark. While powerful, these systems introduced significant operational friction. He describes spending hours aligning Java Virtual Machine versions and deciphering lengthy stack traces for tasks that often did not need massive parallel processing. The complexity of maintaining Spark clusters for routine reporting became a bottleneck rather than an enabler.

Around 2022, Martin discovered DuckDB, an embedded analytical database that runs locally without external dependencies. He found he could query CSV, Parquet, and Iceberg files directly using standard SQL, eliminating the need for cluster management. This simplicity led him to adopt DuckLake, a new table format that stores metadata in a SQL database rather than as numerous small files. Martin reports that catalog operations that previously took seconds with Apache Iceberg now resolve in under a millisecond with DuckLake, significantly speeding up query planning and execution.

Key details

• Martin transitioned from Apache Spark and Google BigQuery to DuckDB to reduce infrastructure overhead for non-petabyte workloads.
• DuckLake stores table metadata in a SQL database, avoiding the performance degradation caused by thousands of small metadata files in formats like Iceberg.
• The upcoming DuckDB v2.0 alpha introduces asynchronous I/O, which Martin benchmarks as 67% to 70% faster for reading large numbers of files from cloud storage.
• Dynamic ATTACH statements in v2.0 allow connection strings and credentials to be pulled from environment variables at runtime, improving security and flexibility.
• The Quack remote protocol enables DuckDB instances to serve data over HTTP, allowing secure access to DuckLake tables without exposing cloud credentials to end users.
• Martin emphasizes that most teams do not need real-time processing for monthly reports, challenging the common assumption that big data tools are always necessary.

Background

Traditional big data architectures often rely on distributed systems like Apache Spark to process data across many servers. These systems excel at handling petabytes of data but require careful configuration, version management, and significant computational resources to start up and run. Metadata management in open table formats like Apache Iceberg involves writing multiple JSON and manifest files for every transaction. As datasets grow, the number of these metadata files increases, slowing down the time it takes for the system to plan queries because it must read and reconcile all these files first.

DuckDB takes a different approach by running as a single process, similar to SQLite but optimized for analytics. It can read data directly from object storage like Amazon S3 without moving it into a proprietary warehouse. DuckLake builds on this by using a relational database to track table metadata. Since relational databases are highly optimized for indexing and querying structured data, looking up table information becomes nearly instantaneous, regardless of how many transactions have occurred. This architecture reduces the operational burden on engineering teams while maintaining high performance for typical analytical queries.

Why it matters

For teams that self-host their software or manage internal data platforms, operational complexity is a major cost driver. Maintaining Spark clusters or managing Iceberg metadata requires specialized knowledge and constant attention. If a tool can deliver the required performance with less infrastructure, it frees up engineering time for feature development rather than platform maintenance. Martin’s experience suggests that many organizations over-engineer their data stacks, paying for scale they never use. Adopting lighter tools like DuckDB can reduce cloud bills and simplify deployment pipelines.

The improvements in DuckDB v2.0 also address common pain points in cloud-native environments. Asynchronous I/O means that reading thousands of log files from S3 no longer blocks the main thread, making local analysis viable for tasks that previously required a distributed cluster. The ability to attach catalogs dynamically using environment variables enhances security by keeping secrets out of code repositories. For IT leads, this means easier compliance and safer automation. The Quack protocol further simplifies access control, allowing teams to expose data services without managing complex identity federation or granting broad cloud permissions.

What you can do

• Audit your current data workloads to identify reports or analyses that do not require real-time processing or petabyte-scale throughput.
• Test DuckDB locally by installing the package and querying a sample CSV or Parquet file to evaluate ease of use against your current stack.
• Experiment with DuckLake by attaching an S3 bucket using the provided SQL commands to compare metadata performance with your existing table format.
• Review your pipeline scripts for hard-coded credentials and plan to migrate to dynamic ATTACH statements once DuckDB v2.0 is stable.
• Consider using the Quack protocol to serve internal dashboards, reducing the need for direct cloud storage access for end users.
• Follow Martin’s Substack for detailed benchmarks and configuration examples if you decide to pilot these tools in your environment.