In this paper, we investigate the behavior of a MDLSTM- RNN architecture to recognize challenging children handwriting in French language. The system is trained across compositions of synthetic adult handwriting and small collections of real children dictations gathered from first classes elementary school. The paper presents the results of investigations concerning handwriting recognition in a context of weak annotated dataset and synthetic images generation for data augmentation. Considering very poor databases of children handwriting, we propose series of experiments to show how the model can cope with small quantity of data. In the paper, we show that assuring a controlled variability of words of varying lengths composed by different instances of degraded or poorly-shaped characters, allows a better generalization of the Handwriting Text Recognition (HTR). We also investigate different choices and splitting propositions to compose both training and validation sets, with respect to children styles distributions. The paper presents conclusions of best suited strategies improving HTR accuracies. Compared to performances to train children real data only, the paper illustrates the impact of transfer learning from adults handwriting (from IAM dataset) and the impact of GAN handwriting styles augmentations to improve chil- dren handwriting recognition. We show in the paper also that data augmentation through scaling, rotation or even repeating same instances of words allows to enhance performances reaching sometimes human level.