The next Galactic core-collapse supernova (CCSN) will be a unique opportunity to study within a fully multi-messenger approach the explosion mechanism responsible for the formation of neutron stars and stellar-mass black holes. State-of-the-art numerical simulations of those events reveal the complexity of the gravitational-wave emission which is highly stochastic. This challenges the possibility to infer the properties of the compact remnant and of its progenitor using the information encoded in the waveforms. In this paper we take further steps in a program we recently initiated to overcome those difficulties. In particular we show how oscillation modes of the proto-neutron star, highly visible in the gravitational-wave signal, can be used to reconstruct the time evolution of their physical properties. Extending our previous work where only the information from a single detector was used we here describe a new data-analysis pipeline that coherently combines gravitational-wave detectors' data and infers the time evolution of a combination of the mass and radius of the compact remnant. The performance of the method is estimated employing waveforms from 2D and 3D CCSN simulations covering a progenitor mass range between 11$\mathrm{M_{\odot}}$ and 40$\mathrm{M_{\odot}}$ and different equations of state for both a network of up to five second-generation detectors and the proposed third-generation detectors Einstein Telescope and Cosmic Explorer. Our study shows that it will be possible to infer PNS properties for CCSN events occurring in the vicinity of the Milky Way, up to the Large Magellanic Cloud, with the current generation of gravitational-wave detectors.