After more than three decades of research in robot manipulation problems, we observed a considerable level of maturity in different related problems. Many high-performant objects pose tracking exists, one of the main problems for these methods is the robustness again occlusion during in-hand manipulation. This work presents a new multimodal perception approach in order to estimate the pose of an object during an in-hand manipulation. Here, we propose a novel learning-based approach to recover the pose of an object in hand by using a regression method. Particularly, we fuse the visual-based tactile information and depth visual information in order to overpass occlusion problems commonly presented during robot manipulation tasks. Our method is trained and evaluated using simulation. We compare the proposed method against different state-of-the-art approaches to show its robustness in hard scenarios. The recovered results show a reliable increment in performance, while they are obtained using a benchmark in order to obtain replicable and comparable results.