Phosphoric acid manufacturing generates large amounts of phosphogypsum (PG); a by-product generally disposed in the surface or evacuated in the seawater without any pretreatment. Phosphogypsum may host non-negligible amounts of valuable elements such as rare earth elements (REEs), which are critical elements on the global market. Surface disposal of PG may be a sustainable option to allow further processing in order to recover valuable elements. However, surface disposal exposes PG to atmospheric conditions (e.g., water, oxygen) which may increase their reactivity and accelerate the release rate of chemical species. This study aims to evaluate the trace element release rate from PG at atmospheric conditions. The studied PG samples were collected from a Moroccan phosphate treatment plant. The samples were characterized for their (i) chemical composition using inductively coupled plasma optical emission spectrometry (ICP-OES) for major elements and inductively coupled plasma mass spectrometry (ICP-MS) for trace elements; (ii) mineralogical composition by X-ray diffraction (XRD), scanning electron microscope equipped with energy-dispersive spectrometer (SEM–EDS), laser-induced breakdown spectroscopy (LIBS), and the mineral chemical composition was analyzed by electron probe microanalyzer (EPMA) and laser ablation inductively coupled plasma mass spectrometry (LA-ICPMS); and (iii) chemical species release rate using leaching tests over 24 h at 25 and 60 °C. Chemically, the PG samples were mainly composed of Ca (23.03–23.35 wt.%), S (17.65–17.71 wt.%), and Si (0.75–0.82 wt.%), and non-negligible amounts of trace elements: REE (344–349 ppm), Cd (3.5–7.4 ppm), U (9.3–27.4 ppm). Mineralogically, the PGs are mainly formed by gypsum (94.2–95.9 wt.%) and quartz (1.67–1.76 wt.%). In terms of chemical species release, the PGs showed a higher reactivity at 60 °C compared to room temperature with a higher release rate at the beginning of the leaching tests. Quantitatively, the PG samples released 3.57–4.11 µg/L/day of REE, 3.18–17.29 µg/L/day of U, and 1.67–5.49 µg/L/day of Cd. Based on the leaching results, we concluded that the trace elements (e.g., U, Cd, REE) are incorporated in PG crystal lattice, which may explain their low concentrations in the leachates. Consequently, total digestion of PG matrix is required to solubilize REE.