This letter studies the problem of autonomous navigation for unmanned underwater vehicles, using computer vision for localization. Parallel tracking and mapping is employed to localize the vehicle with respect to a visual map, using a single camera, whereas an extended Kalman filter (EKF) is used to fuse the visual information with data from an inertial measurement unit, in order to recover the scale of the map and improve the pose estimation. A proportional integral derivative controller controller with compensation of the restoring forces is proposed to accomplish trajectory tracking, where a pressure sensor and a magnetometer provide feedback for depth control and yaw, respectively, while the remaining states are provided by the EKF. Real-time experiments are presented to validate the navigation strategy, using a commercial remotely operated vehicle (ROV), the BlueROV2, which was adapted to perform as an autonomous underwater vehicle with the help of the robot operative system.