Binary neutron stars (BNS) and neutron star-black hole (NSBH) binaries are one of the most promising gravitational wave (GW) sources to probe matter effects. Upcoming observing runs of LIGO-Virgo-KAGRA detectors and future third generation detectors like Einstein Telescope and Cosmic Explorer will allow the extraction of detailed information on these matter effects from the GW signature of BNS and NSBH systems. One subtle effect which may be helpful to extract more information from the detection of compact binary systems is the non-linear memory. In this work, we investigate the observational consequences of gravitational wave non-linear memory in the presence of matter effects. We start by quantifying the impact of non-linear memory on distinguishing BNS mergers from binary black holes (BBH) or NSBH mergers. We find that for the third generation detectors, the addition of non-linear memory to the GW signal model expands the parameter space where BNS signals become distinguishable from the BBH and NSBH signals. Using numerical relativity simulations, we also study the non-linear memory generated from the post-merger phase of BNS systems. We find that it does not show a strong dependence on the equation of state of the NS. However, the non-linear memory from the BNS post-merger phase is much lower than the one from BBH systems of the same masses. Furthermore, we compute the detection prospects of non-linear memory from the post-merger phase of NS systems by accumulating signal strength from a population of BNS mergers for the current and future detectors. Finally, we discuss the impact of possible linear memory from the dynamical ejecta of BNS and NSBH systems and its signal strength relative to the non-linear memory. We find that linear memory almost always has a much weaker effect than non-linear memory.