Our main objective in this work is to address the challenge of enhancing the forecasting of agents trajectories for Connected and Autonomous Vehicles (CAVs) while prioritizing privacy. We introduce an innovative approach to Federated Learning tailored to the contextual aspects of trajectory prediction. We employ the Siamese Neural Network (SNN) to capture context similarities between clients’ environments. Subsequent cluster formation employs SNN to group clients with similar static contexts for federated training, enhancing learning efficiency.Results of our experiments on real-world datasets collected from the highway drone dataset (highD) and the intersection drone dataset (inD) combination, quantified by utilizing wellestablished metrics such as Average Displacement Error (ADE) and Final Displacement Error (FDE), validate the effectiveness of our approach, obtaining superior trajectory prediction capabilities, showcasing the successful alignment of Federated learning with the intricate challenges of trajectory forecasting, all while prioritizing privacy.