An Efficient Approach for Assessing Hyperparameter Importance 论文

摘要

The performance of many machine learning meth-ods depends critically on hyperparameter set-tings. Sophisticated Bayesian optimization meth-ods have recently achieved considerable successes in optimizing these hyperparameters, in several cases surpassing the performance of human ex-perts. However, blind reliance on such methods can leave end users without insight into the rela-tive importance of different hyperparameters and their interactions. This paper describes efficient methods that can be used to gain such insight, leveraging random forest models fit on the data already gathered by Bayesian optimization. We first introduce a novel, linear-time algorithm for computing marginals of random forest predictions and then show how to leverage these predictions within a functional ANOVA framework, to quan-tify the importance of both single hyperparame-ters and of interactions between hyperparameters. We conducted experiments with prominent ma-chine learning frameworks and state-of-the-art solvers for combinatorial problems. We show that our methods provide insight into the relation-ship between hyperparameter settings and perfor-mance, and demonstrate that—even in very high-dimensional cases—most performance variation is attributable to just a few hyperparameters. 1.