Nonlinear photonics integrated at the chip scale opens the path to new applications in an increasing number of fields such as all-optical computing, high bit rate communications on chip, or embedded sensing with frequency combs and super-continuum sources. All these applications require materials having the best trade-off between optical losses, Kerr refractive index, and compatibility with current nanofabrication facilities. Although optimizing the nanofabrication process can minimize linear optical losses to some extent, optimizing the Kerr index of the materials remains challenging because a clear understanding of the link between atomic structure and optical nonlinearities is still missing. This is precisely what this work addresses for chalcogenide glasses based on thin films of Ge-Sb-Se alloys, a promising class of materials fully compatible with large-scale integration technology from the microelectronics industry. By coupling nonlinear Kerr index metrology with ab initio molecular dynamics calculations of amorphous models, this work unveils the unique molecular patterns in these alloys that are responsible for their unusual nonlinear polarizability. This provides for the first time valuable rules for the design of new optical materials with improved Kerr index enabling miniaturization and implementation of future nonlinear photonic devices that can then operate at significantly lower power.