Combining multisensory sources is crucial to interact with our environment, especially for older people who are facing sensory declines. Here, we examined the influence of textured sounds on haptic exploration of artificial textures in healthy younger and older adults by combining a tactile device (ultrasonic display) with synthetized textured sounds. Participants had to discriminate simulated textures with their right index while they were distracted by three disturbing, more or less textured sounds. These sounds were presented as a real-time auditory feedback based on finger movement sonification and thus gave the sensation that the sounds were produced by the haptic exploration. Finger movement velocity increased across both groups in presence of textured sounds (Rubbing or Squeaking) compared to a non-textured (Neutral) sound. While young adults had the same discrimination threshold, regardless of the sound added, the older adults were more disturbed by the presence of the textured sounds with respect to the Neutral sound. Overall, these findings suggest that irrelevant auditory information was taken into account by all participants, but was appropriately segregated from tactile information by young adults. Older adults failed to segregate auditory information, supporting the hypothesis of general facilitation of multisensory integration with aging.