In recent years, there has been an expedited trend in embracing bold and radical innovation of computer architectures, aiming at the continuation of computing performance improvement despite the slowed-down physical device scaling. One new frontier in this field focuses on Artificial Intelligence (AI) hardware. While functionality of AI hardware still remains the main focus, testability and dependability of these new architectures need to be addressed before the mainstream adoption. This survey paper covers the state-of-the-art in research and development of dependability and testability solutions for AI hardware including digital or analog implementations of Artificial Neural Networks (ANNs) and Spiking Neural Networks (SNNs), used in accelerators and neuromorphic designs. Trends, challenges and perspectives are also discussed in this paper.