In this paper, we study the drying of water-saturated porous polydimethylsiloxane (PDMS) elastomers with closed porosity in which the evaporation of water is possible only via the diffusion across the PDMS. Starting from water/PDMS emulsions, we fabricate soft macroporous samples with different pore diameter distributions and average diameters ranging from 10 to 300 µm. In these materials, the drying may lead either to a collapsed state with low porosity or to the cavitation and reopening of a fraction of the pores. Using optical microscopy and porosity measurements, we showed the importance of the pore diameters and interactions on the result of drying. At pore diameters lower than 30 µm, the majority of pores remain collapsed. We attribute the permanence of the collapse of most small pores to a low probability of cavitation and to the adhesion of the pore walls. Pores with diameters larger than 100 µm tend to reopen after the water they contain cavitates. The behavior of pores with diameters ranging from 30 to 100 µm depends on the porosity and drying temperature. We also visualize collective cavitation upon drying of sponges initially saturated with a sodium chloride solution. In this case, the cavitation in the largest pores leads to reopening of small pores in a neighboring zone of the sample. To our knowledge, our results present the first experimental proof of the pore-size-dependent and cooperative nature of the response of soft sponges with closed porosity to drying.