A nonlinear optimal control approach is proposed for the model of the autonomous 3-DOF laboratory helicopter. This UAV model exhibits three degrees of freedom while receiving two control inputs and stands for a benchmark nonlinear and underactuated robotic system. The helicopter's model can perform 3 rotational motions (pitch, roll and yaw turn) while being actuated only by two DC motors. Taylor series expansion around a temporary operating point which is recomputed at each iteration of the control method is used to perform approximate linearization of the dynamic model of the helicopter. The linearization procedure relies on the computation of the Jacobian matrices of the state-space model of the helicopter. Next, an H-infinity feedback controller is designed for the approximately linearized model. The proposed control method provides a solution to the optimal control problem for the nonlinear and multivariable dynamics of the helicopter, under model uncertainties and external perturbations. An algebraic Riccati equation is solved at each time-step of the control method so as to compute the controller's feedback gains. The new nonlinear optimal control approach achieves fast and accurate tracking for all state variables of the 3DOF helicopter, under moderate variations of the control inputs. Finally Lyapunov analysis is used to prove the stability properties of the control scheme.