Negative values of the Casimir entropy occur quite frequently at low temperatures in arrangements of metallic objects. The physical reason lies either in the dissipative nature of the metals as is the case for the plane-plane geometry or in the geometric form of the objects involved. Examples for the latter are the sphere-plane and the sphere-sphere geometry, where negative Casimir entropies can occur already for perfect metal objects. After appropriately scaling out the size of the objects, negative Casimir entropies of geometric origin are particularly pronounced in the limit of large distances between the objects. We analyze this limit in terms of the different scattering channels and demonstrate how the negativity of the Casimir entropy is related to the polarization mixing arising in the scattering process. If all involved objects have a finite zero-frequency conductivity, the channels involving transverse electric modes are suppressed and the Casimir entropy within the large-distance limit is found to be positive.