Variance Reduction Through Bootstrap Ensemble Aggregation

via “variance-reduction through bootstrap ensemble aggregation”

* 🏆 1998: [Gradient-based learning applied to document recognition (CNN/GTN)](https://ieeexplore.ieee.org/abstract/document/726791)

Unique: Introduces bootstrap resampling (sampling with replacement) as a principled mechanism to create diverse training sets for ensemble members, enabling variance reduction without requiring base learner modification or access to additional data — a novel approach in 1996 that differs from prior ensemble methods by leveraging statistical resampling theory rather than algorithmic manipulation

vs others: Simpler and more general than boosting (no sequential weighting or adaptive resampling required) and applicable to any base learner, but less effective at bias reduction than boosting and only beneficial for unstable predictors unlike boosting's broader applicability

via “ensemble-based multi-class classification with bootstrap aggregation”

* 🏆 2001: [A fast and elitist multiobjective genetic algorithm (NSGA-II)](https://ieeexplore.ieee.org/abstract/document/996017)

Unique: Uses random feature subsets at each split (not just random samples) to decorrelate trees, combined with maximum-depth growth and no pruning — this specific combination of randomization sources (data + features) is more effective at variance reduction than single-source randomization used in earlier ensemble methods

vs others: Outperforms single decision trees by 10-30% on typical tabular datasets due to variance reduction through decorrelation, while remaining faster to train than gradient boosting methods and requiring less hyperparameter tuning than neural networks