The Internet of Things is taking hold in our everyday life. Regrettably, the security of IoT devices is often being overlooked. Among the vast array of security issues plaguing the emerging IoT, we decide to focus on access control, as privacy, trust, and other security properties cannot be achieved without controlled access. This article classifies IoT access control solutions from the literature according to their architecture (e.g., centralized, hierarchical, federated, distributed) and examines the suitability of each one for access control purposes. Our analysis concludes that important properties such as auditability and revocation are missing from many proposals while hierarchical and federated architectures are neglected by the community. Finally, we provide an architecture-based taxonomy and future research directions: a focus on hybrid architectures, usability, flexibility, privacy, and revocation schemes in serverless authorization.