Generative models based on latent random variables are a popular tool for time series forecasting. Generative models include the Hidden Markov Model, the Recurrent Neural Network and the Stochastic Recurrent Neural Network. In this paper, we exploit the Pairwise Markov Models, a generalization of Hidden Markov models, as generative models. We first show that the previous generative models are a particular instance of Pairwise Markov models. Next, we also show that they can potentially model a large class of distributions for given observations. In particular, we analyze the particular linear and Gaussian case, where it is possible to characterize the modeling power of these generative models. Finally, we present a parameter estimation algorithm for general Pairwise Markov Models based on Bayesian variational approaches. Simulations are presented and support our statements.