We study a deep linear network expressed under the form of a matrix factorization problem. It takes as input a matrix $X$ obtained by multiplying $K$ matrices (called factors and corresponding to the action of the layers). Each factor is obtained by applying a fixed linear operator to a vector of parameters satisfying a constraint. The number of factors is not limited. In machine learning, the error between the product of the estimated factors and $X$ (i.e. the reconstruction error) relates to the statistical risk. In this paper, we provide necessary and sufficient conditions on the network topology under which stable recovery holds. This means that the error on the parameters defining the factors (i.e. the stability of the recovered parameters) scales linearly with the reconstruction error (i.e. the risk). Therefore, under these conditions on the network topology, any successful learning task leads to stably defined features and therefore interpretable layers/network. In order to do so, we first evaluate how the Segre embedding and its inverse distort distances. Then, we show that any deep linear network can be cast as a generic multilinear problem (that uses the Segre embedding). We call this method {\em tensorial lifting}. Using the tensorial lifting, we provide necessary and sufficient conditions for the identifiability of the factors (up to a scale rearrangement). We finally provide the necessary and sufficient condition called \NSPlong~(because of the analogy with the usual Null Space Property in the compressed sensing framework) which guarantees that the stable recovery of the factors holds. We illustrate the theory with a practical example where the deep linear network is a convolutional linear network. As expected, the conditions are rather strong but not empty. A simple test on the network topology can be implemented to test if the condition holds.