The spatiotemporal organization of chromatin influences many nuclear processes: from chromosome segregation to transcriptional regulation. To get a deeper understanding of these processes, it is essential to go beyond static viewpoints of chromosome structures, to accurately characterize chromatin's diffusion properties. We present GP-FBM: a computational framework based on Gaussian processes and fractional Brownian motion to extract diffusion properties from stochastic trajectories of labeled chromatin loci. GP-FBM uses higher-order temporal correlations present in the data, therefore, outperforming existing methods. Furthermore, GP-FBM allows to interpolate incomplete trajectories and account for substrate movement when two or more particles are present. Using our method, we show that average chromatin diffusion properties are surprisingly similar in interphase and mitosis in mouse embryonic stem cells. We observe surprising heterogeneity in local chromatin dynamics, correlating with potential regulatory activity. We also present GP-Tool, a user-friendly graphical interface to facilitate usage of GP-FBM by the research community.