We show that the carrier "antibinding" observed recently in semiconductor quantum dots, i.e., the fact that the ground state energy of two electron-hole pairs goes above twice the ground-state energy of one pair, can entirely be assigned to a charge separation effect, whatever its origin. In the absence of external electric field, this charge separation comes from different "spreading-out" of the electron and hole wavefunctions linked to the finite height of the barriers. When the dot size shrinks, the two-pair energy always stays below when the barriers are infinite. On the opposite, because barriers are less efficient for small dots, the energy of two-pairs in a dot with finite barriers, ends by behaving like the one in bulk, i.e., by going above twice the one-pair energy when the pairs get too close. For a full understanding of this "antibinding" effect, we have also reconsidered the case of one pair plus one carrier. We find that, while the carriers just have to spread out of the dot differently for the "antibinding" of two-pairs to appear, this "antibinding" for one pair plus one carrier only appears if this carrier is the one which spreads out the less. In addition a remarkable sum rule exists between the "binding energies" of two pairs and of one pair plus one carrier.