Following some recent proposals to handle natural language generation in spoken dialogue systems with long short-term memory recurrent neural network models (Wen et al., 2016a), this work first investigates a variant thereof with the objective of a better integration of the attention sub-network. Our second objective is to propose and evaluate a framework to adapt the NLG module on-line through direct interactions with users. The basic way to do so is to lead the users to utter alternative sentences rephrasing the expression of a particular dialogue act. To add such a new sentence to its model, the system can rely on automatic transcription, which is costless but error-prone, or ask the user to transcribe it manually, which is almost flawless but costly. To optimise this choice, we investigate a reinforcement learning approach based on an adversarial bandit scheme. The bandit reward is defined as a linear combination of expected payoffs, on the one hand, and costs of acquiring the new data provided by the user, on the other hand. We show that this definition allows the system designer to find the right balance between improving the system performance, for a better match with the user's preferences, and limiting the burden associated with it. Finally, the actual benefits of the system are assessed through a human evaluation, showing that the progressive inclusion of more diverse utterances increases user satisfaction.