Following our previous results on Markov chains on periodic graphs, we study the convergence in rough path topology of a certain class of discrete processes, the hidden Markov walks, to a Brownian motion with an area anomaly. This area anomaly, which is a new object keeps track of the time-correlation of the discrete models and brings into light the question of embeddings of discrete processes into continuous time. We also identify an underlying combinatorial structure in the hidden Markov walks, which turns out to be a generalization of the occupation time from the classical ergodic theorem in the spirit of rough paths. Finally, we construct a rough path out of iterated sums of a hidden Markov walk, compare it to the classical canonical lift of the piecewise linear interpolation of the process and analyze its convergence to a non-geometric rough path.