In this paper, we propose LUVI, Lightweight UWB-VIO relative positioning method for indoor localization. Recent designs of handheld and embedded devices feature various technologies which have the means to enhance localization performance in indoor environments. These include visual odometry based on cameras and augmented reality, and communication hardware such as UWB. Integration of such technologies to exploit their advantages allows us to compensate for each other's errors in measurement. This improves the overall function of future services, such as visual representation of sensing information from sensors in areas that are not physically visible. However, existing work cannot fully exploit these technologies to high extent, often inducing high errors or wasted resources. LUVI is a novel localization method which estimates the location of a target object using relative coordinates of estimator devices without the aid of definitive coordinates. LUVI focuses on utilization of lightweight management of virtual anchors for localization, with functions that reduce the computing and communication complexity while maintaining the accuracy and improving energy efficiency of the localization. Our work has been fully implemented and tested in several indoor environments, showing robustness to NLOS while significantly reducing computational complexity, and up to 30% lower average error.