When performing artificial intelligence, CPUs and GPUs consume considerably more energy for moving data between logic and memory units than for doing arithmetic. Brains, by contrast, achieve superior energy efficiency by fusing logic and memory entirely. Currently, emerging memory nanodevices give us an opportunity to reproduce this concept. In this overview paper, we look at neuroscience inspiration to extract lessons on the design of memory-centric neuromorphic systems. We study the reliance of brains on approximate memory strategies, which can be reproduced for AI. We give the example of a hardware binarized neural network with resistive memory. Based on measurements on a hybrid CMOS/resistive memory chip, we see that such systems can exploit the properties of emerging memories without error correction, and achieve extremely high energy efficiency. Second, we see that brains use the physics of their memory devices in a way much richer than only storage. This can inspire radical electronic designs, where memory devices become a core part of computing. We have for example fabricated neural networks where magnetic memories are used as nonlinear oscillators to implement neurons, and their electrical couplings implement synapses. Such designs can harness the rich physics of nanodevices, without suffering from their drawbacks.