A classical random walk (St; t 2 N) is defined by St := Xt n=0 Xn, where (Xn) are i.i.d. When the increments (Xn)n2N are a one-order Markov chain, a short memory is introduced in the dynamics of (St). This so-called "persistent" random walk is nolonger Markovian and, under suitable conditions, the rescaled process converges towards the integrated telegraph noise (ITN) as the time-scale and space-scale parameters tend to zero (see [11, 17, 18]). The ITN process is effectively non-Markovian too. The aim is to consider persistent random walks (St) whose increments are Markov chains with variable order which can be infinite. This variable memory is enlighted by a one-to-one correspondence between (Xn) and a suitable Variable Length Markov Chain (VLMC), since for a VLMC the dependency from the past can be unbounded. The key fact is to consider the non Markovian letter process (Xn) as the margin of a couple (Xn;Mn)n 0 where (Mn)n 0 stands for the memory of the process (Xn). We prove that, under a suitable rescaling, (Sn;Xn;Mn) converges in distribution towards a time continuous process (S0(t);X(t);M(t)). The process (S0(t)) is a semi-Markov and Piecewise Deterministic Markov Process whose paths are piecewise linear.