Internet Memes (IMs) are creative media that combine text and vision modalities that people use to describe their situation by reusing an existing, familiar situation. Prior work on IMs has focused on analyzing their spread over time or high-level classification tasks like hate speech detection, while a principled analysis of their stratified semantics is missing. Hypothesizing that Semantic Web technologies are appropriate to help us bridge this gap, we build the first Internet Meme Knowledge Graph (IMKG): an explicit representation with 2 million edges that capture the semantics encoded in the text, vision, and metadata of thousands of media frames and their adaptations as memes. IMKG is designed to fulfil seven requirements derived from the inherent characteristics of IMs. IMKG is based on a comprehensive semantic model, it is populated with data from representative IM sources, and enriched with entities extracted from text and vision connected through background knowledge from Wikidata. IMKG integrates its knowledge both in RDF and as a labelled property graph. We provide insights into the structure of IMKG, analyze its central concepts, and measure the effect of knowledge enrichment from different information modalities. We demonstrate its ability to support novel use cases, like querying for IMs that are based on films, and we provide insights into the signal captured by the structure and the content of its nodes. As a novel publicly available resource, IMKG opens the possibility for further work to study the semantics of IMs, develop novel reasoning tasks, and improve its quality.