vaex

Implements a deferred computation model where DataFrame operations (e.g., df.x * df.y) are stored as expression trees rather than executed immediately. Virtual columns are calculated on-the-fly during materialization, avoiding intermediate memory allocation. The expression system defers actual computation until results are explicitly needed (visualization, aggregation, export), enabling efficient processing of billion-row datasets by processing only required data chunks.

Leverages OS-level memory mapping (mmap) to map data files directly into virtual address space, loading only accessed data pages into physical RAM on-demand. The DataFrame abstraction sits atop memory-mapped datasets (via dataset_mmap.py), enabling transparent access to files larger than available memory. Zero-copy operations mean column slicing and filtering create views rather than copies, with the kernel handling page faults and eviction automatically.

Implements a comprehensive data type system supporting numeric (int, float, complex), string, datetime, boolean, and categorical types with automatic inference from source data. Type conversion is lazy (deferred until materialization) and supports explicit casting via expressions. The system handles missing values (NaN, None) appropriately for each type. Array conversion to NumPy/Arrow formats is optimized for zero-copy where possible.

Provides vectorized string operations (substring, split, replace, case conversion, pattern matching) implemented in C++ for performance. String operations work on virtual columns without materializing intermediate results. The system supports regular expressions and Unicode handling. Operations are lazy and composed into expression trees for efficient batch processing.

Implements efficient statistical aggregations (sum, mean, std, min, max, median, percentiles, etc.) computed in a single pass over the data using Welford's algorithm and other numerically stable techniques. Aggregations work on virtual columns and support filtering and grouping. Results are computed lazily and materialized only when needed. The system maintains numerical stability for large datasets.

Provides sorting capabilities using external memory techniques (merge sort with disk spillover) for datasets larger than RAM. Sorting operations create ordered views or materialized sorted DataFrames. The system supports sorting on multiple columns with mixed sort orders (ascending/descending). Sorting is lazy when possible but may require materialization for certain operations. Index-based access enables efficient lookups on sorted data.

Provides export functionality to HDF5, Apache Arrow, Apache Parquet, CSV, and other formats with automatic format selection based on use case. Export operations materialize data and write to disk with optional compression. The system supports incremental export (appending to existing files) and format conversion. Export can be parallelized across multiple threads for improved throughput.

Implements a task-based execution model (via execution.py and tasks.py) where deferred expressions are compiled into tasks that execute on thread pools. The engine batches operations, manages task dependencies, and coordinates multithreaded execution across CPU cores. Tasks operate on chunked data, allowing efficient parallelization while respecting memory constraints. Progress tracking and cancellation are built into the execution pipeline.

Implements efficient group-by operations using hash-based binning rather than sorting, allowing O(n) aggregations without requiring data to be pre-sorted. The GroupBy abstraction supports multiple aggregation functions (sum, mean, count, std, etc.) computed in a single pass over the data. Virtual columns enable grouping on derived expressions without materializing intermediate results. Results are returned as new DataFrames with group keys and aggregated values.

Implements multiple join strategies (hash join, sort-merge join) selected based on data characteristics and memory availability. The join operation builds hash tables or sorts data as needed, supporting inner, left, right, and outer joins. Joins operate on DataFrames with automatic alignment of join keys, and results are returned as new DataFrames. The system optimizes join order and strategy selection based on dataset size and cardinality.

Provides unified import interface supporting HDF5, Apache Arrow, Apache Parquet, CSV, and JSON formats with automatic format detection and optimization recommendations. The system includes format-specific dataset classes (e.g., HDF5Dataset, ArrowDataset) that implement memory-mapped access where possible. CSV/JSON require full materialization but are automatically converted to optimized formats for repeated access. The import pipeline handles compression, encoding, and type inference.

Provides interactive visualization capabilities through a server-based architecture (vaex-server) that streams aggregated data to browser-based frontends. The visualization system computes histograms, heatmaps, and scatter plots on the server side, sending only aggregated results to the client. This enables interactive exploration of billion-row datasets with responsive UI updates. The server handles query execution, caching, and result streaming.

Provides wrapper classes for scikit-learn, XGBoost, and other ML frameworks that integrate with Vaex's lazy evaluation system. Features can be engineered as virtual columns without materialization, and models are trained on materialized data only when needed. The system supports feature scaling, encoding, and transformation pipelines that operate on expressions. Model predictions can be added back as virtual columns for further analysis.

Implements a multi-level caching system that stores computed results (aggregations, filtered views, materialized columns) with automatic invalidation when source data changes. The cache tracks dependencies between operations, invalidating only affected cached results when mutations occur. Cache eviction policies balance memory usage with hit rates. The system supports both in-memory and disk-based caching for large intermediate results.

Provides efficient row filtering through boolean indexing and selection operations that create lazy views without materializing filtered data. Selections can be combined using boolean operators (AND, OR, NOT) and chained for complex filtering logic. The system supports filtering on both materialized columns and virtual (derived) columns. Filtered views maintain the original data structure and can be further processed or materialized on demand.