safetensors

Implements a custom binary format (8-byte header + JSON metadata + contiguous data buffer) that eliminates pickle's arbitrary code execution vulnerability by design. The format uses a simple, declarative structure with no dynamic code loading or object reconstruction, making it safe to load from untrusted sources. Validation occurs at the Rust core level (~400 lines) before any Python object instantiation, preventing malicious payloads from executing during deserialization.

Implements memory-mapped file access through the Rust core's safe_open() context manager, which maps the safetensors file directly into process memory without copying tensor data. The JSON header is parsed once to build an offset index, then individual tensors are accessed on-demand by calculating byte offsets into the contiguous data buffer. This approach eliminates the memory overhead of eager loading and enables partial tensor access without materializing the entire model.

Implements jax-specific save_file() and load_file() functions that handle JAX array conversion, including jax.Array dtype mapping, shape preservation, and device-agnostic loading (arrays are loaded on the default JAX device). The adapter extracts raw array data from JAX arrays, passes to Rust core for serialization, and reconstructs JAX arrays on load. This enables JAX/Flax-based workflows to use safetensors without framework-specific code.

Implements mlx-specific save_file() and load_file() functions that handle MLX tensor conversion, including mlx.core.array dtype mapping, shape preservation, and Apple Silicon device handling. The adapter extracts raw tensor data from MLX arrays, passes to Rust core for serialization, and reconstructs MLX arrays on load. This enables MLX-based workflows (optimized for Apple Silicon) to use safetensors without framework-specific code.

Provides command-line and Python API utilities for converting models from other formats (PyTorch .pt, TensorFlow SavedModel, HuggingFace Transformers) to safetensors format. The conversion process loads the source model using framework-specific APIs, extracts the tensor dictionary, and serializes using safetensors. This is implemented as a set of utility functions in the Python bindings that abstract framework-specific loading logic.

Implements on-demand tensor slicing through the safe_open() context manager, which parses the JSON header to compute byte offsets for each tensor, then allows slice operations (e.g., tensor[0:100, :]) to be resolved without loading the full tensor. The slicing logic calculates the exact byte range needed based on tensor shape, dtype, and requested indices, then reads only that range from the file. This is implemented in the Rust core's slice.rs module (~270 lines) and exposed through Python bindings.

Provides a unified serialization API that abstracts framework differences through framework-specific adapter modules (torch, numpy, tensorflow, jax, mlx). Each adapter implements save_file() and load_file() functions that convert framework tensors to/from a common internal representation before writing to the safetensors binary format. The Rust core handles the actual serialization; Python adapters handle dtype mapping, device placement, and framework-specific tensor construction. This design enables a single .safetensors file to be loaded by any supported framework.

Encodes tensor metadata (shape, dtype, data type, byte offset) in a compact JSON header that is parsed once at file open time. The JSON structure maps tensor names to metadata objects containing shape arrays, dtype strings (e.g., 'F32', 'I64'), and byte offsets into the data buffer. This metadata enables the Rust core to validate tensor consistency, compute slice offsets, and construct framework-specific tensors without scanning the data buffer. The header is limited to 100MB to prevent DOS attacks.

Provides save_file(tensors_dict, filepath) and load_file(filepath) functions that serialize/deserialize entire tensor dictionaries in a single operation. The save_file() function iterates over the dictionary, computes cumulative byte offsets for each tensor, builds the JSON header with metadata, and writes the contiguous data buffer. The load_file() function reads the header, parses metadata, and returns a dictionary of framework-specific tensors. This API abstracts the complexity of offset calculation and buffer management.

Enforces strict validation rules at the Rust FFI boundary to prevent denial-of-service attacks: header size is capped at 100MB, header must begin with '{' character (0x7B), and all tensor offsets are validated against file size before any data access. The validation occurs before Python object construction, preventing malicious files from consuming excessive memory or triggering expensive operations. This is implemented in the Rust core's validation logic (~100 lines).

Implements torch-specific save_file() and load_file() functions that handle PyTorch tensor conversion, including dtype mapping (torch.float32 → 'F32'), device handling (GPU tensors are moved to CPU before serialization), and gradient state management (requires_grad is not preserved). The adapter uses PyTorch's tensor API to extract raw data and metadata, then passes to the Rust core for serialization. On load, tensors are constructed as CPU tensors and can be moved to device explicitly.

Implements numpy-specific save_file() and load_file() functions that handle NumPy array conversion, including dtype mapping (np.float32 → 'F32'), shape preservation, and byte order handling (little-endian assumed). The adapter extracts raw array data and metadata using NumPy's C API, passes to Rust core for serialization, and reconstructs NumPy arrays on load. This enables NumPy-based workflows to use safetensors without framework-specific code.

Implements tensorflow-specific save_file() and load_file() functions that handle TensorFlow tensor conversion, including tf.Variable to tensor conversion, dtype mapping (tf.float32 → 'F32'), and shape preservation. The adapter extracts raw tensor data from TensorFlow variables, passes to Rust core for serialization, and reconstructs TensorFlow tensors on load. This enables TensorFlow-based workflows to use safetensors without framework-specific code.