We systematically study the interesting relations between the quantum elliptic Calogero-Moser system (eCM) and its generalization, and their corresponding supersymmetric gauge theories. In particular, we construct the suitable characteristic polynomial for the eCM system by considering certain orbifolded instanton partition function of the corresponding gauge theory. This is equivalent to the introduction of certain co-dimension two defects. We next generalize our construction to the folded instanton partition function obtained through the so-called “gauge origami” construction and precisely obtain the corresponding characteristic polynomial for the doubled version, named the elliptic double Calogero-Moser (edCM) system.