We study a deep/multi-layer structured matrix factorization problem. It approximates a given matrix by the product of K matrices (called factors). Each factor is obtained by applying a fixed linear operator to a short vector of parameters (thus the name " structured "). We call the model deep or multi-layer because the number of factors is not limited. In the practical situations we have in mind, we typically have K = 10 or 20. We provide necessary and sufficient conditions for the identifiability of the factors (up to a scale rearrangement). We also provide a sufficient condition that guarantees that the recovery of the factors is stable. A practical example where the deep structured factorization is a convolutional tree is provided in an accompanying paper.