Polars

Polars defers execution of DataFrame operations by building an expression IR (intermediate representation) that is analyzed by a query optimizer before physical execution. The optimizer performs predicate pushdown (filtering before joins), column pruning (selecting only needed columns), and redundant computation elimination. This is implemented via the polars-plan crate which constructs a logical plan from the expression DSL, then converts it to an optimized physical plan executed by either the streaming or memory engine.

Polars stores all data in Apache Arrow columnar format (via polars-arrow crate), organizing data by column rather than row. This enables SIMD vectorization, better CPU cache locality, and efficient compression. The columnar layout allows zero-copy data sharing between Polars and other Arrow-compatible libraries (DuckDB, Pandas with PyArrow backend, Apache Spark). Data is stored in ChunkedArray structures that can reference the same underlying memory buffers.

Polars implements comprehensive temporal operations (via polars-time crate) including timezone-aware datetime handling, date/time arithmetic, duration operations, and temporal component extraction (year, month, day, hour, etc.). Temporal types are stored efficiently as integer offsets from epoch, with timezone information preserved. Operations like date addition, duration calculation, and timezone conversion are vectorized.

Polars uses PyO3 (Python-Rust FFI framework) to expose the Rust core to Python without data copying. The FFI layer (in polars-python crate) marshals Python objects to Rust data structures and vice versa, leveraging Arrow's memory layout for zero-copy interop. Python DataFrames are thin wrappers around Rust DataFrame objects, with method calls dispatched to Rust implementations via PyO3 bindings.

Polars implements categorical (enum) data type with dictionary encoding, storing unique values once and referencing them by integer index. This reduces memory usage for columns with many repeated values and accelerates grouping/joining operations. The categorical system (in polars-core crate) supports both ordered and unordered categories, with optional specification of category order.

Polars supports nested data types including Struct (named fields), List (variable-length arrays), and Array (fixed-length arrays) with recursive operations that can navigate and transform nested structures. The type system (polars-core crate) allows operations like unnesting (flattening), extracting fields, and applying expressions to nested values. Nested types are stored efficiently in Arrow format with proper memory layout.

Polars provides Node.js bindings (via polars-python crate compiled to WASM or native modules) enabling JavaScript/TypeScript developers to use Polars with async/await semantics. The bindings expose the same API as Python but adapted for JavaScript conventions. Async operations allow non-blocking data processing in Node.js event loop.

Polars implements two physical execution engines: a streaming engine that processes data in chunks without loading entire datasets into memory, and a memory engine for operations requiring full dataset visibility (e.g., sorts, joins). The query optimizer selects which engine to use based on operation type and available memory. The streaming engine processes data in configurable batch sizes (default 1M rows), enabling processing of datasets larger than RAM.

Polars provides a Python/Node.js DSL for building expressions (pl.col(), pl.lit(), pl.when(), etc.) that are type-checked against the DataFrame schema before execution. The expression system (polars-plan crate) performs automatic type coercion (e.g., int to float for division) and validates that operations are valid for the column types. Schema inference happens during lazy frame construction, catching type errors before data is processed.

Polars implements format-specific I/O handlers (CSV, Parquet, NDJSON, JSON, Avro, ORC, database connectors) in the polars-io crate with intelligent predicate pushdown to the storage layer. For Parquet files with Hive partitioning, Polars can filter partitions before reading data. For database sources, filter predicates are pushed down as SQL WHERE clauses. The I/O layer integrates with the query optimizer to apply filters before data enters memory.

Polars includes a SQL parser (polars-sql crate) that translates SQL queries into the native expression DSL, enabling users to write SQL while benefiting from Polars' optimization and execution engines. The SQL interface supports standard SQL operations (SELECT, WHERE, JOIN, GROUP BY, window functions) and translates them to the same logical plan used by the expression API, ensuring identical performance and optimization.

Polars implements efficient grouped aggregation via the GroupBy API, which partitions data by key columns and applies aggregation expressions to each group. The aggregation engine (in polars-ops crate) supports both standard functions (sum, mean, count, etc.) and custom expressions, with automatic parallelization across groups. Multiple aggregations can be applied in a single pass over the data, reducing memory allocations.

Polars implements multiple join algorithms (hash join, sort merge join, nested loop join) and automatically selects the optimal strategy based on data size and cardinality. The join optimizer (in polars-plan crate) can reorder joins to minimize intermediate result sizes and push predicates before joins. Supported join types include inner, left, right, full outer, cross, and anti/semi joins with optional coalesce of duplicate key columns.

Polars implements window functions (row_number, rank, dense_rank, lag, lead, first, last, sum over window, etc.) via the expression system with support for partitioning (OVER clause) and ordering. Window functions are computed efficiently by the polars-ops crate, which partitions data, applies the window function within each partition, and maintains row order. Supports both unbounded and bounded windows (e.g., last 7 rows).

Polars provides a comprehensive string API (via polars-ops crate) supporting regex operations (match, extract, replace, split), pattern matching (contains, starts_with, ends_with), case conversion, trimming, and Unicode-aware operations. String operations are vectorized and can be applied to entire columns efficiently. Regex patterns are compiled once and reused across the column.