We study the minimal recurrent spiking neural network of a single neuron with an autaptic synapse. We implement the neural system in the solid state with a recently introduced ultracompact neuron (UCN) model, which is based on the memristive properties of a thyristor. The UCN is supplemented by a self-synaptic, autaptic, connection, where we control the feedback. Both excitatory and inhibitory cases are considered. We explore the systematic behavior as a function of autaptic intensity and feedback time delay. We realize a tunable dynamic memory, showing graded persistent activity, where short excitatory and inhibitory pulses allow the firing rate to be controlled. We finally reproduce recent experimentally observed behavior of a biological autapse measured in vivo, finding excellent qualitative agreement. Our work opens the way into the field of solid-state neuroscience, with the UCN as an accessible platform to implement and experimentally study the dynamic behavior of spiking neural networks.