This paper aims at providing statistical guarantees for a kernel-based estimation of time-varying parameters driving the dynamic of infinite memory processes introduced by Doukhan and Wintenberger \cite{DW}. We then extend the results of Dahlhaus {\it et al.} \cite{DRW} on local stationary Markov processes to other important models such as the GARCH model. The estimators are computed as localized M-estimators of any contrast satisfying appropriate regularity conditions. % as in Bardet and Wintenberger \cite{BW}. We prove the uniform consistency and the pointwise asymptotic normality of such kernel-based estimators. We apply our results to usual contrasts such as least-square, least absolute value, or quasi-maximum likelihood contrasts. Various time-varying models such as AR$(\infty$), ARCH$(\infty)$ and LARCH$(\infty)$ are considered. We discuss their approximation of locally stationary ARMA and GARCH models under contraction conditions. Numerical experiments demonstrate the efficiency of the estimators on both simulated and real data sets.