Mining big data to make predictions or decisions is the main goal of modern artificial intelligence (AI) and machine learning (ML) applications. Vast innovation in algorithms, their software implementations and data management has enabled great progress to date, but wide adoption has been slowed by limited capabilities of existing computing hardware. The low communication bandwidth between memory and processing units in conventional von Neumann machines does not support the requirements of emerging applications that rely extensively on large sets of data. More recent computing paradigms, such as high parallelization and near-memory computing (e.g., in GPUs) help alleviate the data communication bottleneck to some extent, but paradigm-shifting concepts are required. In-memory computing has emerged as a prime candidate to eliminate this bottleneck by co-locating the memory and processing. In this context, resistive switching (RS) memory devices is a key promising choice, due to their unique intrinsic device-level properties enabling both storing and computing with a small, massively-parallel footprint at a low power. Theoretically, this directly translates to a major boost in energy efficiency and computational throughput, but various practical challenges remain. We present a qualitative and quantitative analysis of several key existing challenges in implementing high-capacity, high-volume RS memories for accelerating the most computationally demanding computation in ML inference – that of vector-matrix multiplication (VMM). Monolithic integration of RS memories with CMOS integrated circuits is presented as the core underlying technology. We review key existing design choices in terms of device-level physical implementation, circuit-level design, and system-level considerations, and provide an outlook for future directions.