The impressive increasing availability of social media posts has given rise to considerable research challenges. This article is concerned with the problem of semantic location prediction of geotagged tweets. The underlying task is to associate to a social media post, the focal spatial object, if any (e.g., Place Of Interest POI), it topically focuses on. Although relevant for a number of applications such as POI recommendation, this problem has not so far received the attention it deserves. In previous work, the problem has mainly been tackled by means of language models that rely on costly probability estimation of word relevance across spatial regions. We propose the Spatially-aware Geotext Matching (SGM) model, which relies on a neural network learning framework. The model combines exact word-word-local interaction matching signals with semantic global tweet-POI interaction matching signals. The local interactions are built over kernel spatial word distributions that allow revealing spatially driven word pair similarity patterns. The global interactions consider the strength of the interaction between the tweet and the POI from both the spatial and semantic perspectives. Experimental results on two real-world datasets demonstrate the effectiveness of our proposed SGM model compared to state-of-the-art baselines including language models and traditional neural interaction-based models.