DuckDB

Executes SQL queries directly on Parquet, CSV, and JSON files using a columnar vectorized execution engine that processes data in SIMD-friendly chunks (DataChunk vectors) without materializing entire datasets into memory. The engine uses the Vector and DataChunk abstraction layer from the type system to enable cache-efficient batch processing of billions of rows, with lazy evaluation and predicate pushdown to minimize I/O.

Automatically infers Parquet file schemas and applies filter predicates at the file-reading layer to skip row groups and columns that don't match query conditions. Uses the Parquet Integration module to parse metadata without reading full column data, enabling sub-millisecond filtering decisions on multi-terabyte datasets. Supports nested type handling via the Variant Type system for complex Parquet structures.

Provides the Query Profiler System that captures detailed execution metrics (operator timing, row counts, memory usage) for each query operator. Integrates with the Logging Infrastructure to record profiling data and enable performance analysis. Supports both per-query profiling and aggregate statistics across multiple queries.

Implements the Sorting, Scanning, and Execution Pipeline with multiple sort strategies (in-memory quicksort, external merge sort with spilling). The scanning layer supports both full table scans and index-based scans with filter pushdown. Uses the Buffer Management layer to handle memory pressure during sorting operations, automatically spilling to disk when necessary.

Provides an in-process database engine that can operate in both memory-only mode (for ephemeral analysis) and persistent mode (with data stored in DuckDB's native format). Uses the Storage Engine with row groups and column data organization to maintain data durability while preserving columnar format. Supports both read-only and read-write modes with configurable access patterns.

Integrates with Apache Arrow's Inter-Process Communication (IPC) format to enable zero-copy data exchange with other Arrow-compatible systems (Pandas, Polars, PyArrow, R, etc.). Uses Arrow RecordBatch as the internal representation, allowing data to be shared across language boundaries without serialization. Supports both reading and writing Arrow IPC files and streaming Arrow data.

Implements a comprehensive type system that includes scalar types (INTEGER, VARCHAR, TIMESTAMP) and nested types (STRUCT for objects, LIST for arrays, MAP for key-value pairs). Nested types can be arbitrarily nested and are stored efficiently in columnar format. The type system integrates with the query planner and optimizer, enabling type-aware optimizations and function overload resolution.

Implements hash join operations with configurable execution modes (build-probe, semi-join, anti-join) using the Hash Join Implementation pattern. The engine selects join strategies based on table sizes and available memory, with support for both in-memory hash tables and spilling to disk when memory pressure exceeds configured thresholds. Uses the Buffer Management and Compression layer to manage memory efficiently during large joins.

Executes aggregate functions (SUM, COUNT, AVG, MIN, MAX, GROUP_CONCAT, etc.) and window functions (ROW_NUMBER, RANK, LAG, LEAD, etc.) using vectorized operators that process DataChunk batches. Supports both simple aggregations and complex window frames with partition-by and order-by clauses. Uses the Aggregation and Window Functions operator pattern with streaming aggregation for memory efficiency.

Applies the Optimizer Pipeline to transform logical query plans into optimized physical plans using cost-based heuristics. The Join Order Optimization module evaluates different join orderings and selects the lowest-cost plan based on estimated row counts and cardinality. Uses the Binder System to validate query semantics and the Table Catalog Management to resolve table references and statistics.

Reads CSV files with automatic type inference and configurable parsing options (delimiter, quote character, escape character, header detection). Uses the CSV Reading and Writing module to handle edge cases like quoted fields with embedded delimiters and newlines. Supports streaming reads for files larger than memory via the Multi-File Reader pattern, with parallel file reading for multiple CSV inputs.

Parses and queries JSON data (line-delimited JSON, nested objects, arrays) using the JSON Extension. Supports JSON path expressions for extracting nested fields and converting JSON to relational tables. Uses the Variant Type system to represent JSON values natively, enabling SQL operations on semi-structured data without explicit schema definition.

Provides a Python API that integrates seamlessly with Pandas and Polars DataFrames, enabling zero-copy data exchange via Arrow IPC format. Supports both eager execution (returning results immediately) and lazy evaluation (building query plans). The Python API wraps the C API and uses Arrow RecordBatch as the internal representation for efficient data transfer.

Provides an Extension Architecture that allows developers to register custom functions, types, and table functions via the Function Registry and Autoloading system. Extensions are loaded dynamically at runtime and integrated into the query optimizer and execution engine. Supports both built-in extensions (JSON, Delta, Parquet) and user-defined extensions via C++ or SQL.

Implements transaction support via the Table Storage and Transactions module, providing ACID guarantees for INSERT, UPDATE, and DELETE operations. Uses row-group versioning and write-ahead logging to ensure durability and isolation. Supports both read-committed and serializable isolation levels with configurable transaction behavior.