Significance: Intraoperative optical imaging is a localization technique for the functional areas of the human brain cortex during neurosurgical procedures. These areas can be assessed by monitoring cerebral hemodynamics and metabolism. Robust quantification of these biomarkers is complicated to perform during neurosurgery due to the critical context of the operating room. In actual devices, the inhomogeneities of the optical properties of the exposed brain cortex are poorly taken into consideration, which introduce quantification errors of biomarkers of brain functionality. Moreover, the best choice of spectral configuration is still based on an empirical approach. Aim: We propose a digital instrument simulator to optimize the development of hyperspectral systems for intraoperative brain mapping studies. This simulator can provide realistic modeling of the cerebral cortex and the identification of the optimal wavelengths to monitor cerebral hemodynamics (oxygenated HbO 2 and deoxygenated hemoglobin Hb) and metabolism (oxidized state of cytochromes b and c and cytochrome-c-oxidase oxCytb, oxCytc, and oxCCO). Approach: The digital instrument simulator is computed with white Monte Carlo simulations of a volume created from a real image of exposed cortex. We developed an optimization procedure based on a genetic algorithm to identify the best wavelength combinations in the visible and near-infrared range to quantify concentration changes in HbO 2 , Hb, oxCCO, and the oxidized state of cytochrome b and c (oxCytb and oxCytc). Results: The digital instrument allows the modeling of intensity maps collected by a camera sensor as well as images of path length to take into account the inhomogeneities of the optical properties. The optimization procedure helps to identify the best wavelength combination of 18 wavelengths that reduces the quantification errors in HbO 2 , Hb, and oxCCO by 47%, 57%, and 57%, respectively, compared with the gold standard of 121 wavelengths between 780 and 900 nm. The optimization procedure does not help to resolve changes in cytochrome b and c in a significant way but helps to better resolve oxCCO changes. Conclusions: We proposed a digital instrument simulator to optimize the development of hyperspectral systems for intraoperative brain mapping studies. This digital instrument simulator and this optimization framework could be used to optimize the design of hyperspectral imaging devices.