A rubberlike matrix highly filled with spherical micrometric glass beads is submitted to uniaxial tension tests until break. X-ray tomography imaging performed on the material while submitted to uniaxial tension reveals early debonding at the matrix/filler interfaces at the poles of the particles followed by void coalescence creating damage localization. The latter causes a downturn of the macroscopic stress-strain response. These phenomena are analyzed further with three-dimensional finite element simulations, where 64 spherical beads are distributed randomly in a periodic cell. A simple version of the Tvergaard-Hutchinson cohesive-zone model allows to reproduce all the experimental trends well. The effects of the three parameters involved are analyzed, and three different types of macroscopic behaviors are observed corresponding to three different microstructure damages. The value of the initial stiffness of the interface, limited by numerical convergence, has little effect on how the local damage evolves but has a significant impact on the overall macroscopic stress values. The local damage is strongly dependent on the critical strength and the separation failure displacement, and the adhesion energy may be considered as a resulting parameter of the two previous ones. The interfacial critical strength appears to have a significant impact on the damage initiation, either spread across the structure for low values, or localized for high values. Increasing the interface separation failure displacement delays the possible loss of adhesion to a higher strain and preserves the integrity of the composite material.