Smart home IoT devices lack proper security, raising safety and privacy concerns. One-size-fits-all network administration is ineffective because of the diverse QoS requirements of IoT devices. Device classification can improve IoT administration and security. It identifies vulnerable and rogue items and automates network administration by device type or function. Considering this, a promising research topic focusing on Machine Learning (ML)-based traffic analysis has emerged in order to demystify hidden patterns in IoT traffic and enable automatic device classification. This study analyzes these approaches to understand their potential and limitations. It starts by describing a generic workflow for IoT device classification. It then looks at the methods and solutions for each stage of the workflow. This mainly consists of i) an analysis of IoT traffic data acquisition methodologies and scenarios, as well as a classification of public datasets, ii) a literature evaluation of IoT traffic feature extraction, categorizing and comparing popular features, as well as describing open-source feature extraction tools, and iii) a comparison of ML approaches for IoT device classification and how they have been evaluated. The findings of the analysis are presented in taxonomies with statistics showing literature trends. This study also explores and suggests undiscovered or understudied research directions.