Talk on "High-Dimensional Bayesian Optimization with a Combination of Kriging Models"
Résumé
Kriging models, also called Gaussian Process regression, are commonly used as surrogate models of expensive computer codes in various applications. In engineering design optimization, Bayesian approaches based on Kriging models such as efficient global optimization (EGO) are employed to speed-up the optimization process by reducing the number of function evaluations. These methods have been successfully applied to many real-world applications in low dimensions (less than 30 design parameters).
However, engineering designs can often be parametrized by more than 50 parameters in practice. In higher dimensions, Kriging suffers from the curse of dimensionality and building an accurate surrogate model is met with various setbacks. One of the main challenges is related to optimizing the covariance length-scale hyperparameters of the model. These hyperparameters regulate the decay of the correlation between observations when their distance increases, and anisotropic Kriging models consider one length-scale per dimension. Estimating the length-scales correctly is essential to obtain a model with a good accuracy. They are typically obtained by maximizing the log-likelihood of the model. In high-dimension, this inner optimization is problematic due to the exponential growth of the search space with the dimension and to over-fitting issues when there are too few observations. For these reasons, maximum likelihood estimation of the hyperparameters often fails to provide correct values, especially when not enough observations are available. This is problematic for Bayesian optimization where the number of samples at the start of the optimization is low, and where inaccurate surrogates negatively impact the convergence speed. Several papers address these difficulties by either reducing the dimension of the problem, or by considering simplifying hypothesis such as additive models. However, both these methods make additional assumption on the underlying function to approximate (low dimension representation or additive structure) which are not necessarily satisfied in practice, thus they are not easily generalizable to any design engineering problems.
In this presentation, we introduce a new method for high-dimensional Bayesian optimization using a convex combination of Kriging models. This method bypasses the length-scales optimization by combining sub-models with random length-scales and its expression is obtained in closed-form expression avoiding any inner optimization. We also describe how to sample suitable length-scales for the sub-models using a criterion based on the entropy of the correlations, in order to avoid degenerated sub-models with either too large or too small length-scales. Finally, the variance of the combination being not directly available as the correlations between sub-models are unknown, we present a method to compute the prediction variance for any weighting methods. This is done by introducing a global covariance structure also based on a linear combination of covariances. We apply our combined Kriging model to high-dimensional EGO for analytical test functions and for the design of an electric machine. We show that the classical Kriging approach using maximum likelihood estimation fails to properly optimize the length-scale hyperparameters and that our method successfully builds more accurate surrogate models at the beginning of the optimization loop. This results in faster convergence speed for EGO using the combination, thus reducing the number of computer code evaluations to reach the optimum.