This paper is devoted to the convergence analysis of stochastic approximation algorithms of the form theta(n+1) = theta(n) + gamma H-n+1(theta n) (Xn+1), where {theta(n), n is an element of N} is an R-d-valued sequence, {gamma(n), n is an element of N} is a deterministic stepsize sequence, and {X-n, n is an element of N} is a controlled Markov chain. We study the convergence under weak assumptions on smoothness-in-theta of the function theta bar right arrow H-theta(x). It is usually assumed that this function is continuous for any x; in this work, we relax this condition. Our results are illustrated by considering stochastic approximation algorithms for (adaptive) quantile estimation and a penalized version of the vector quantization.