Humans learn to both visually recognize individual objects and categorize them at different levels of abstraction. Such multi-semantic representation is crucial to efficiently reason about the world. However, it is currently unclear how such representations could be learned with the very sparse labeling available to human learners. To answer this question we let an artificial agent play with objects while occasionally "hearing" their category label. Our agent assigns similar representations to a) similarly labelled and b) close-in-time visual inputs. We show that our agent learns a 2-level hierarchical representation that first aggregates different views of objects and then brings together different objects to form categories. Interestingly, we do not observe a trade-off between each semantic content. Our work suggests that the temporal structure of visual experience during object play together with occasional labeling suffice for learning a hierarchically structured object/category representation.