The computer analysis of Whole Slide Images (WSIs) is becoming increasingly prevalent in pathology-based diagnosis, although its analysis presents considerable challenges due to the voluminous nature of the data. To address this issue, Multiple Instance Learning (MIL) has emerged
as a viable approach that involves partitioning WSIs into tiles for processing. Nevertheless, conventional MIL methodologies inadequately capture the essential spatial context between tiles, which is imperative for accurate diagnosis across various diseases. In parallel, non-MIL, spatial-aware methods have also emerged, primarily based on adapting convolutional neural network architectures to the high-resolution setting. However, these methods are generally GPU memory-intensive, restricting the architecture's size and scalability. In this paper, we present a novel framework, SparseXceptionMIL (SparseXMIL), aiming to enhance the GPU efficiency of spatial interactions modeling within WSI data through the introduction of a multidimensional sparse image representation and a novel pooling operator. By integrating sparse convolutions within the Xception architecture, this operator enables efficient spatial information modeling at both local and global scales. Empirical evaluations conducted on various classification tasks, encompassing subtyping for breast and lung carcinomas and predicting abnormalities in the DNA damage response in breast cancer WSI, consistently demonstrate the advantages of our approach over benchmark methods in terms of accuracy and GPU efficiency. These results underscore the potential of sparse convolutional architectures to improve WSI classification while ensuring minimal GPU memory usage. Our experiments' source code is available at https://github.com/loic-lb/SparseXMIL.