Evaluating the Accuracy of Ensemble Machine Learning and Statistical Uncertainty: Spatial Prediction of Topsoil Thickness in Iowa

Abstract

The objectives of this study were to assess spatial predictions of topsoil thickness from models produced from ensemble machine learning algorithms along with assessing the uncertainty estimations associated with those models. Boosting is one example of an ensemble method, which attempts to improve accuracy by combining multiple weaker models into a unified, stronger model. The inherent problem with ensemble learning is that model accuracy is evaluated with subsets of the sample population, which can result in model overfitting. Cross-validation (CV) partitions the sample population into multiple random training and validation subsets. The training data calibrate a model and test prediction accuracy on the validation subset. To determine if a model is truly robust, accuracy should be assessed by testing model predictions on an independent validation (IV) subset. To evaluate model accuracy in this study, over 900 samples from central Iowa were used for model training and over 600 digital terrain derivatives were generated as predictor variables to develop mathematical models with the machine-learning algorithm, Cubist. Fifty-four observations were selected from the three townships and reserved for independent validation. Three ensemble methods, 1) bootstrapped-bagged 2) bootstrapped-boosted, and 3) 10 fold cross-validation (CV)-boosted were applied to determine which method was most robust. A 90% confidence interval was calculated from ten prediction realizations of the bootstrapped-bagged method to determine how many IV observations were contained within the estimated range of uncertainty.

The CV-boosted method was most robust in prediction with an IV relative error of 46% (Mean Absolute Error (MAE) = 26.5 cm). The bootstrapped-bagged and bootstrapped-boosted models had IV relative errors of 83% and 51%, respectively. For validation of the prediction interval, 53% of the IV observations were contained within the range of uncertainty and the average interval width was 51 cm. Maps of statistical uncertainty revealed that the model confidently predicts topsoil thickness in upland soil-landscapes and performs poorly in fluvial and pluvial areas.