The explosive growth of in-memory computing and neural network requires stringent demands on the computational energy efficiency. Nonvolatile memories such as magnetic random access memory (MRAM) provides alternative memory solutions toward energy efficiency. Sparsity realization across emerging device, hybrid circuit, and algorithmic becomes a recent trend in neural network. Previous sparse adaption in memories mainly focused on high level analysis. In this article, the sparse realization of hybrid magnetic/CMOS integration is first proposed for convolutional neural network (CNN). Simulation results with representative data sets CIFAR-10 show that MRAM sensing operation can be speedup 6.4× with 84.46% sparsity. The proposed training and retraining phases can solve unreliable sensing issues with a proper sparsity selection.