This paper focuses on the quantitative analysis of deep neural networks used in data-driven modeling of induction motor dynamics. With the availability of a large amount of data generated by industrial sensor networks, it is now possible to train deep neural networks. Recently researchers have started exploring the usage of such networks for physics modeling, online control, monitoring, and fault prediction in induction motor operations. We consider the problem of estimating speed and torque from currents and voltages of an induction motor. Neural networks provide quite good performance for this task when analysed from a machine learning perspective using standard metrics. We show, however, that there are some caveats in using machine learning metrics to analyze a neural network model when applied to induction motor problems. Given the mission-critical nature of induction motor operations, the performance of neural networks has to be validated from an electrical engineering point of view. To this end, we evaluate several traditional neural network architectures and recent state of the art architectures on dynamic and quasi-static benchmarks using electrical engineering metrics.