State-space models (SSM) and recurrent neural networks (RNN) are widely used approaches for dynamical system modeling. In the case of SSMs, they include explicit modeling of all components, including the noise characterization, and thus allow for interpretability and uncertainty quantification. However, the underlying dynamical model parameters need to be specified and closed-form inference is possible only in a few simple cases. RNNs, on the other hand, can learn, through supervised training, rather complex nonlinearities from the data but lack the aforementioned advantages of SSMs. In this work, we combine the benefits of both approaches by introducing a Gaussian SSM whose state and evolution operators can be learnt from the data. In order to deal with the ill-posedness of this parameter estimation problem, we propose an innovative factorized form of both the state and observation operators, reminiscent from deep nonnegative matrix factorization models. An expectationmaximization method combined with a block alternating strategy is introduced to estimate each of the involved positive latent factors, while jointly performing the probabilistic state inference. Our resulting formulation and inference tool is called deep recurrent dictionary learning (DRDL). We then specialize DRDL for the problem of stock forecasting, by proposing an online training strategy and a probabilistic assessment of the trading decision. Numerical experiments on a problem of stock market data inference shows its superiority among several state-of-theart dynamic modeling tools.