The adaptive immune system is a diverse ecosystem that responds to pathogens by selecting cells with specific receptors. While clonal expansion in response to particular immune challenges has been extensively studied, we do not know the neutral dynamics that drive the immune system in the absence of strong stimuli. Here, we learn the parameters that underlie the clonal dynamics of the T cell repertoire in healthy individuals of different ages, by applying Bayesian inference to longitudinal immune repertoire sequencing (RepSeq) data. Quantifying the experimental noise accurately for a given RepSeq technique allows us to disentangle real changes in clonal frequencies from noise. We find that the data are consistent with clone sizes following a geometric Brownian motion and show that its predicted steady state is in quantitative agreement with the observed power-law behavior of the clone-size distribution. The inferred turnover time scale of the repertoire increases with patient age and depends on the clone size in some individuals.