In this paper, we propose Multi-Scale Group Convolution (MSGC) an optimization to the conventional convolutional layer, to address the high computational complexity issue in deploying convolutional neural networks (CNN) on the Internet of Things (IoT) enabled edge sensors. The proposed method reduces complexity by grouping input channels of a convolution layer into smaller groups, thereby reducing the number of intermediate connections and complexity of matrix computations in a CNN. This approach results in a minor performance loss, which is compensated by utilizing a characteristic of group convolution to extract multi-scale features. The proposed technique is applied for detecting cardiac arrhythmias from electrocardiogram (ECG) data using CNNs to be deployed in edge sensors. For the binary classification of ECG into Normal or Anomalous beats, the proposed MSGC-based CNN achieved an average 30% reduction in computations while achieving similar or better performance compared to the conventional CNNs. We used the Physionet MIT-BIH Arrhythmia database for performance evaluation, and in the best scenario, our approach increases accuracy by 0.47%, F1 score by 1.87% while only using 64.41% MACs and 83.62% parameters. This optimization strategy can be extended to other CNN models where computational complexity reduction is critical for deployment in edge devices.