Talk on: "Bayesian Optimization for High-Dimensional Problems Using a Combination of Kriging Surrogate Models"
Résumé
Numerical optimization has been widely used to solve design engineering problems in recent years. However, due to the large number of numerical simulations required, the optimization process involved is often computationally expensive. In order to accelerate the optimization process, strategies based on surrogate models are often used. One such approach, efficient global optimization (EGO), based on Kriging surrogate models, has been successfully applied to a number of real-worlds applications in low-dimensions (less than 20).
However, in engineering design optimization, the designs are often parametrized by more than 50 shape parameters. The ordinary Kriging method scales poorly for high-dimensional problems, and building a good Kriging model is met with various setbacks. One of the main challenges is related to length-scale hyperparameter optimization. Most Kriging models consider one length-scale hyperparameter per dimension which all need to be optimized simultaneously. This is typically done by maximizing the log-likelihood of the model. For high-dimensional problems, this optimization is problematic due to the exponential growth of the search space with the dimension, to the shape of the log-likelihood function and to its computational cost, and to over-fitting issues when there are too few observations. Several papers address these difficulties by reducing the dimension of the problem, for instance by embedding the design space into a lower dimensional space, or by considering simplifying hypothesis such as additive models.
In this presentation, we propose a new method for high-dimensional Kriging models which bypasses the length-scales optimization by combining Kriging sub-models with fixed length-scales. Contrarily to other approaches, this does not rely on dimension reduction and preserves the correlation between all input variables. Furthermore, the combination has a closed-form expression and does not require any inner optimization. We also describe a novel approach to obtain suitable bounds for the length-scales of the sub-models, and we compare different weighting schemes for the approximation of high-dimensional test functions. Finally, we present a way to obtain the variance of the combination for any weighting methods and we apply our combined model to high-dimensional EGO. We show that the classical Kriging approach using maximum likelihood estimation fails to properly optimize the length-scale hyperparameters and that our combination of sub-models with fixed length-scales successfully build more accurate surrogate models for EGO.