We take dissipation into account in the derivation of the Casimir energy formula between two objects placed in a surrounding medium. The dissipation channels are considered explicitly in order to take advantage of the unitarity of the full scattering processes. We demonstrate that the Casimir energy is given by a scattering formula expressed in terms of the scattering amplitudes coupling internal channels and taking dissipation into account implicitly. We prove that this formula is also valid when the surrounding medium is dissipative.