The computer analysis of Whole Slide Images (WSI) is becoming increasingly prevalent in pathology-based diagnosis, although it presents considerable challenges due to the voluminous nature of the data. To address this issue, Multiple Instance Learning (MIL) has emerged as a viable approach that involves partitioning WSI into tiles for processing. Nevertheless, conventional MIL methodologies inadequately capture the essential spatial context between tiles, which is imperative for accurate diagnosis across various diseases. In this paper, we present a novel framework, SparseXceptionMIL (\myMIL), aiming to enhance the modeling of spatial interactions within WSI data by introducing a multi-dimensional sparse image representation and a novel pooling operator. This operator, integrating sparse convolutions within the Xception architecture, enables effective spatial information processing across multiple scales. Empirical evaluations conducted on various classification tasks, encompassing subtyping for breast and lung carcinomas and predicting abnormalities in the DNA damage response in breast cancer WSI, consistently demonstrate the superiority of our approach over benchmark methods. These results underscore the potential of sparse convolutional architectures to improve WSI classification. The source code for our experiments is made available at https://github.com/loic-lb/SparseXMIL.