Among the hypotheses relating dyslexia to a temporal processing disorder, Hari and Renvall (Hari, R., Renvall, H., 2001. Impaired processing of rapid stimulus sequences in dyslexia. Trends. Cognit. Sci. 5, 525-532.) argued that dyslexic individuals would show difficulties at an attentional level, through sluggish attentional shifting (SAS) in all sensory modalities. However, the amodality assumption of the SAS theory was never straightforwardly assessed in the same group of dyslexic participants using similar paradigms in both the visual and auditory modalities. Here, the attentional sequential performance of control and dyslexic participants was evaluated using rapid serial presentation paradigms measuring individual stream segregation thresholds in the two modalities. The first experiment conducted on French dyslexic children with a phonological disorder revealed an SAS only in the auditory modality only which was strongly related to reading performance. The second experiment carried out on British dyslexic young adults with a phonological disorder using the same auditory segregation task but a different visual paradigm revealed an SAS in both the visual and the auditory modalities. In addition, a relationship was found in this group between SAS, poor reading and poor phonological skills. Two further control experiments showed that differences in task design or participants' language between Experiments 1 and 2 could not account for the differences in terms of visual segregation patterns. Overall, our results support the view that the auditory SAS plays a role in developmental dyslexia via its impact on phonological abilities. In addition, a visual temporal disorder in dyslexia might emerge at a later developmental stage, when the visual system normally becomes more expert at rapid temporal processing.