This paper presents a novel framework for object detection in videos that considers both structural and temporal information. Detec- tion is performed by first applying low-level feature extraction techniques in each frame of the video. Then, additional robustness is obtained by considering the temporal stability of videos, using particle filters and probability maps, which encode information about the expected location of each object. Lastly, structural information of the scene is described using graphs, which allows us to further improve the results. As a prac- tical application, we evaluate our approach on table tennis sport videos databases: the UCF101 table tennis shots and an in-house one. The ob- served results indicate that the proposed approach is robust, showing a high hit rate on the two databases.