Deep Neural Networks (DNNs) are increasingly used in safety critical autonomous systems. We present MOZART+, a DNN accelerator architecture which provides fault detection and fault tolerance. MOZART+ is a systolic architecture based on the Output Stationary (OS) data-flow, as it is a data-flow that inherently limits fault propagation. In addition, MOZART+ achieves fault detection with on-line functional testing of the Processing Elements (PEs). Faulty PEs are swiftly taken off-line with minimal classification impact. We show how to handle the case of layers with a small number of neurons. The implementation of our approach on Squeezenet results in a loss of accuracy of less than 3% in the presence of a single faulty PE, compared to 15-33% without mitigation. The area overhead for the test logic does not exceed 8%. Dropout during training further improves fault tolerance, without a priori knowledge of the faults. We present a detailed fault-injection study on multiple systolic architectures, considering different fault-models and comparing different measures of accuracy.