Task Mapping And Dynamic Parallelization With Parameter Expansion

via “multi-device-parallelization-with-pmap”

Google's numerical computing library — autodiff, JIT, vectorization, NumPy API for ML research.

Unique: JAX's pmap integrates with jit and grad — @jit @pmap @grad enables a single compiled function that computes gradients in parallel across devices with automatic all-reduce for gradient averaging. pmap is implemented as a tracer that replicates the function across devices and inserts collective communication primitives, enabling seamless composition with other transformations.

vs others: Simpler than explicit distributed training frameworks (Horovod, DeepSpeed) because it requires no manual communication code; more efficient than parameter servers because it uses collective operations and avoids centralized bottlenecks