In this paper we propose a novel approach for multiple object tracking in structured sports videos using graphs. The objects are tracked by combining particle filter and frames description with Attributed Relational Graphs. We start by learning a probabilistic structural model graph from annotated images and then using it to evaluate and correct the current tracking state. Different from previous studies, our approach is also capable of using the learned model to generate new hypotheses of where the object is likely to be found after situations of occlusion or abrupt motion. We test the proposed method on two datasets: videos of table tennis matches extracted from YouTube and badminton matches from the ACASVA dataset. We show that all the players are successfully tracked even after they occlude each other or when there is a camera cut.