The minimal theory of quasidilaton massive gravity with or without a Horndeski-type kinetic term for the quasidilaton field propagates only three physical modes: the two massive tensor polarizations and one scalar mode. This reduced number of degrees of freedom is realized by a Lorentz symmetry violation at cosmological scales and the presence of appropriate constraints that remove unwanted modes. Vacuum cosmological solutions have been considered in a previous work, and it has been shown that the late-time de Sitter attractor is stable under inhomogeneous perturbations. In this work, we explore the stability of cosmological solutions in the presence of matter fields. Assuming for simplicity that the quasidilaton scalar is on an attractor at the level of the background, we derive stability conditions in the subhorizon limit and find the scalar sound speeds, as well as the modification with respect to general relativity to the gravitational potential in the quasistatic approximation. We also find that the speed limit of gravitational waves coincides with the speed of light for any homogeneous and isotropic cosmological background, on or away from the attractor.