Deep learning has revolutionized many image processing tasks such as classification or segmentation and, more recently, gives very promising results for solving inverse problems. It however remains a gap between the deep learning blackbox approaches and the more recently unrolled deep learning techniques proposed to bring the physics of the model and standard solving techniques into the network design. In order to understand more precisely the mechanisms, we place ourselves in the framework of the simple study of image denoising and we study four networks designed from unrolled forward-backward iterations in the dual, FISTA in the dual, Chambolle-Pock, and Chambolle-Pock exploiting the strong convexity. Performance and stability obtained with each of these networks will be detailed. A comparison within these approaches, standard penalized likelihood approaches, and the state-of-the-art black-box approach DnCNN is also provided.