This paper is devoted to the convergence analysis of stochastic approximation algorithms of the form $\theta_{n+1} = \theta_n + \gamma_n H(\theta_n,X_{n+1})$ where $\{{\theta_n,n \in \nset\}$ is a $\rset^d$-valued sequence, {γn, n ∈ N} is a deterministic step-size sequence and {Xn, n ∈ N} is a controlled Markov chain. We study the convergence under weak assumptions on smoothness-in-θ of the function $\theta \to H(\theta,x)$. It is usually assumed that this function is continuous for any x; in this work, we relax this condition. Our results are illustrated by considering stochastic approximation algorithms for (adaptive) quantile estimation and a penalized version of the vector quantization.