Brown dwarfs (BDs) are celestial objects representing the link between the least massive main-sequence stars and giant gas planets. In the first part of this article, we perform a model-independent search of a gamma-ray signal from the direction of nine nearby BDs in 13 years of Fermi-LAT data. We find no significant excess of gamma rays, and we, therefore, set 95 % confidence level upper limits on the gamma-ray flux with a binned-likelihood approach. In the second part of the paper, we interpret these bounds within an exotic mechanism proposed for gamma-ray production in BDs: If the dark matter (DM) of the universe is constituted of particles with non-negligible couplings to the standard model, BDs may efficiently accumulate them through scatterings. DM particles eventually thermalise, and can annihilate into light, long-lived, mediators which later decay into photons outside the BD. Within this framework, we set a stacked upper limits on the DM-nucleon elastic scattering cross section at the level $\sim 10^{-38}$ cm$^{2}$ for DM masses below 10 GeV. Our limits are comparable to similar bounds from the capture of DM particles in celestial objects, but have the advantage of covering a larger portion of the parameter space in mediator decay length and DM mass. They also depend only on the local DM abundance, as opposed to the inner Galaxy profile, and are thus more robust.