Federated learning (FL) is a distributed learning paradigm enabling participants to collaboratively train a machine learning (ML) model. In security-oriented tasks, FL can be used to share attack knowledge, without sharing participants’ local data. Recent research results reveal that highly heterogeneous data distributions can prevent federations from converging towards an appropriate global model. Moreover, maintaining trustworthiness is challenging, as FL-based collaborative intrusion detection systems (CIDSs) are vulnerable to malicious updates. In this demonstration paper, we present critical examples of these challenges using a set of standardized public datasets and a dedicated automation tool. We review the impact of heterogeneity using different data-distribution, before looking at a scenario with malicious actors