An infinite word has the property $R_m$ if every factor has exactly $m$ return words. Vuillon showed that $R_2$ characterizes Sturmian words. We prove that a word satisfies $R_m$ if its complexity function is $(m-1)n+1$ and if it contains no weak bispecial factor. These conditions are necessary for $m=3$, whereas for $m=4$ the complexity function need not be $3n+1$. New examples of words satisfying $R_m$ are given by words related to digital expansions in real bases.