In this letter, we present a visual control framework for accurately positioning feature points belonging to the surface of a 3D deformable object to desired 3D positions, by acting on a set of manipulated points using a robotic manipulator. Notably, our framework considers the dynamic behavior of the object deformation, that is, we do not assume that the object is in its static equilibrium during the manipulation. By relying on a coarse dynamic Finite Element Model (FEM), we have successfully formulated the analytical relationship expressing the motion of the feature points to the six degrees of freedom (6 DOF) motion of a robot gripper. From this modeling step, a novel closed-loop deformation controller is designed. To be robust against model approximations, the whole shape of the object is tracked in real-time using an RGB-D camera, thus allowing to correct any drift between the object and its model on-the-fly. Our model-based and vision-based controller has been validated in real experiments. The results highlight the effectiveness of the proposed methodology.