Optical properties of Nd3+-doped Silicon Rich Silicon Oxide (SRSO) layers grown by magnetron sputtering have been investigated as a function of silicon excess, rare earth content and phase transformation. We observed that, depending on the deposition and annealing conditions, the Nd3+ ions can be excited either via SRSO matrix defects or silicon nanoclusters (Si-nc). The SRSO:Nd3+ layers with optimal photoluminescence intensity were selected and processed in single mode waveguides at 1064nm. Those waveguides were characterized and analysed in terms of losses, guided PL, and gain estimation. In parallel to this experimental work, simulations were carried out by an iterative algorithm based on finite difference time domain method (FDTD) and by the auxiliary differential equations (ADE) method. This leads to a full description of the electromagnetic field (EMF) distribution and the populations of Si-nc excitons and Nd3+ levels. The feasibility of an optical Nd3+-doped amplifying waveguide has been analyzed through the modeling using two pumping configurations: i) top-pumping and ii) co-propagation of pump and probe. The results show that, depending on the Si-nc content, the co-propagation configuration is not always possible due to strong absorption of the pump wave while the top-pumping leads to more homogenous pumping power repartition in the active layer. A net gain reaching some dB.cm-1 at 1064 nm can be expected in Nd-doped waveguides and is correlated with the experimental results.