In this paper, we are concerned with the large N limit of linear combinations of the entries of a Brownian motion on the group of N by N unitary matrices. We prove that the process of such a linear combination converges to a Gaussian one. Various scales of time and initial distribution are concerned, giving rise to various limit processes, related to the geometric construction of the unitary Brownian motion. As an application, we propose a quite short proof of the asymptotic Gaussian feature of the linear combinations of the entries of Haar distributed random unitary matrices, a result already proved by Diaconis et al.