xgboost

Trains gradient boosted decision tree ensembles using a column-block sparse matrix format and level-wise tree growth strategy. XGBoost implements a custom tree-building algorithm that evaluates all possible splits in parallel across features, using weighted quantile sketching to handle large datasets that don't fit in memory. The framework supports both exact greedy splitting and approximate histogram-based splitting with configurable precision tradeoffs.

Performs inference on trained models using GPU acceleration via CUDA/ROCm or CPU fallback, with support for batch prediction on large datasets. XGBoost's prediction engine loads the compiled tree ensemble into GPU memory and evaluates all samples in parallel across the tree structure, achieving 10-100x speedup over CPU inference depending on batch size and tree depth. Supports both single-sample and vectorized batch prediction with automatic device selection.

Assigns different weights to training samples, enabling handling of imbalanced datasets, cost-sensitive learning, and sample importance weighting. XGBoost's training loop incorporates sample weights into gradient/Hessian computation, allowing the model to focus on high-weight samples. Supports both per-sample weights (for importance weighting) and per-class weights (for class imbalance), with automatic weight normalization.

Exports trained trees to human-readable formats (DOT, JSON, text) and visualizes tree structure for model interpretation. XGBoost's plot_tree() function renders individual trees as directed acyclic graphs showing split decisions, leaf values, and sample counts. Exported trees can be visualized in external tools (Graphviz) or analyzed programmatically, enabling debugging and understanding of model behavior.

Extracts multiple types of feature importance scores from trained tree ensembles: gain (average loss reduction per feature), cover (average number of samples affected), and frequency (number of times feature appears in splits). XGBoost traverses the compiled tree structure and aggregates statistics across all trees, supporting both global importance (across entire model) and per-tree importance for interpretability. Importance scores are normalized and can be exported for visualization or downstream analysis.

Allows users to define custom loss functions (objectives) and evaluation metrics via Python callbacks, enabling optimization for domain-specific tasks beyond standard classification/regression. XGBoost's training loop calls user-provided gradient/Hessian functions at each boosting iteration, allowing arbitrary differentiable objectives (e.g., custom ranking losses, fairness-constrained objectives). Custom metrics are evaluated on validation sets and used for early stopping without modifying core training logic.

Monitors evaluation metrics on a held-out validation set during training and stops boosting when validation performance plateaus or degrades, preventing overfitting. XGBoost evaluates the model on validation data after each boosting round, tracks the best metric value, and halts training if no improvement occurs within a configurable patience window (e.g., 10 rounds). Early stopping integrates with custom metrics and supports both single and multi-metric monitoring.

Distributes training across multiple machines using Rabit (XGBoost's custom distributed communication framework) or external schedulers (Spark, Dask, Kubernetes). XGBoost partitions data across nodes, performs local tree construction in parallel, and synchronizes tree updates via allreduce operations, enabling near-linear scaling on large clusters. Supports both data parallelism (different samples on each node) and feature parallelism (different features on each node) with automatic load balancing.

Performs k-fold cross-validation with automatic stratification for classification tasks, evaluating model performance across multiple train/test splits. XGBoost's cv() function partitions data into k folds, trains k models in parallel (one per fold), evaluates each on its held-out fold, and aggregates results (mean and standard deviation of metrics). Supports both stratified (preserves class distribution) and random splitting with custom fold generators.

Saves trained models to disk in multiple formats (native XGBoost binary, JSON, text) and loads them for inference or continued training. XGBoost's save_model() and load_model() functions serialize the entire tree ensemble including hyperparameters, feature names, and metadata, enabling model versioning and deployment across environments. Supports both Python pickle (for full Python objects) and language-agnostic formats (JSON, binary) for cross-platform compatibility.

Integrates with hyperparameter optimization frameworks (Optuna, Ray Tune, Hyperopt) via standard Python APIs, enabling automated search over learning rate, tree depth, regularization, and other parameters. XGBoost's cv() and train() functions return metrics that optimization frameworks use as objectives, supporting both grid search and Bayesian optimization without custom integration code. Supports early stopping within optimization loops to avoid wasting compute on unpromising hyperparameter combinations.

Supports multi-class classification (>2 classes) and multi-output regression (predicting multiple targets simultaneously) via softmax and multi-task learning objectives. XGBoost trains separate tree ensembles for each class/output, sharing the same feature space but learning independent split decisions per class. Predictions return probability distributions (for classification) or multiple regression outputs, enabling complex prediction tasks beyond binary classification.