The present study assessed the effectiveness of the ECRIMO educational application designed to build first‐grade level spelling skills. We tested whether using the app to teach spelling would be as effective as providing the same training using traditional paper exercises. The effect of integrating gamification into mobile learning apps, which has been little studied in the context of young children, is also investigated. A pretest/training/posttest design was implemented with 311 first‐graders divided in four groups: no training, paper training, the ECRIMO app with gamification features, and the ECRIMO app without gamification. Spelling, reading and phonological awareness abilities was measured at both pretest and posttest. The training was conducted over a 7‐week period (4.40 hours in total). The experimental design allowed us to answer three questions: (1) Is spelling training effective regardless of the medium used? (2) Is training through the app as efficient as paper‐based training? (3) Does gamification impact students' learning performance? Mixed‐model analyses revealed (1) a positive effect on the training outcome depended on the initial spelling ability of participants, (2) a comparable efficiency between autonomous training using the ECRIMO app on tablets and the same training provided by teachers using paper exercises and (3) a marginally positive effect of gamification that is greater for the weakest students. The present study proposes an original and pertinent experimental design to test the relevance of educational applications. The design features of learning apps can impact students' learning differently depending on their initial level. A critical step should be verifying that using online apps for training is at least as effective as the same training using paper exercises. Practitioner notes What is already known about this topic A significant number of children experience difficulties in reading and spelling from the first years of learning. The use of new technologies to support classroom teaching is rapidly developing as a topic of interest for educational professionals and researchers. Evaluations of new technologies developed to enhance literacy skills suggest that many factors can vary their effectiveness. The effectiveness of a digital educational application can be enhanced or undermined by design choices, such as gamification. What this paper adds Spelling training with the app ECRIMO seems effective for first year students, especially those with the lowest and middle level. Comparable effects of both the tablet‐based and paper equivalent training on participants' spelling were found. The use of gamification in ECRIMO could be more suitable for the weakest students. Implications for practice and/or policy Educational technologies should be evidence‐based and should be evaluated with both a passive and an active control group. The design should be carefully considered and tested, as it may be advantageous for some students and disadvantageous for others. The use of digital technology in education can be beneficial for classroom practice, when the activity can be carried out in total autonomy, leaving the teacher available for a group of pupils with specific needs.