More and more location-based services become relying on the logical notion of a physical location, known as logical location (e.g. Starbucks, KFC). In this paper, we propose a new way to identify logical location using (1) a GPS-enabled mobile phone and (2) a wearable camera embedded in user's glasses. When a user with a wearable camera is detected paying attention to a certain physical location, all the logical locations within the error range of the GPS coordinates are considered as the matched candidates. We select the representative frames in the video stream corresponding to user's interested location in real-time and use multi-view images taken beforehand to represent each logical location. We then extract the Scale Invariant Feature Transform visual features from both the representative video frames and pre-stored images of candidate logical locations for video-image matching, the logical location that the user pays attention to can thus be identified. In order to differentiate the cases where users watch certain objects rather than a logical location in the street, we use Support Vector Machine to classify the two cases so that only the valid logical location is identified. Our proposed approach is proved weather and user independent, and it does not request additional user efforts compared with previous solutions. The results tested using a real-world dataset can achieve an average accuracy of 91.08%.