In this paper, an AR (Augmented reality) system based on a multimodal user interface is proposed and its core techniques are described. The objective of this work is to realize certain task that operators can use the virtual robot to ameliorate the interaction with the real world using AR environment. VC++ and OpenGL are used to create this application. AR applications (ARToolKit) is used to detect the real-time position and orientation of camera and to facilite the operator’s observation and operation. Experiments prove that the system is reliable and efficient. Simulation result is obtained and it is very useful for designers to ameliorate the interaction with the real world.