An inertial-aided visual servo control approach for fully-actuated Autonomous Underwater Vehicles (AUVs) without relying on linear velocity measurements is proposed. The homography obtained from corresponding images of a locally planar scene is directly exploited as feedback information. A cascade inner-outer loop control architecture is adopted that facilitates both control implementation and gain tuning procedures. Control robustness with respect to model uncertainties and external disturbances is reinforced using both high gain estimation and integrator techniques. Rigourous stability analysis and insightful discussions are provided throughout the paper. Finally, robustness and efficiency of the proposed approach are illustrated by simulation on a realistic AUV model.