We define a tau function for a generic Riemann-Hilbert problem posed on a union of non-intersecting smooth closed curves with jump matrices analytic in their neighborhood. The tau function depends on parameters of the jumps and is expressed as the Fredholm determinant of an integral operator with block integrable kernel constructed in terms of elementary parametrices. Its logarithmic derivatives with respect to parameters are given by contour integrals involving these parametrices and the solution of the Riemann-Hilbert problem. In the case of one circle, the tau function coincides with Widom's determinant arising in the asymptotics of block Toeplitz matrices. Our construction gives the Jimbo-Miwa-Ueno tau function for Riemann-Hilbert problems of isomonodromic origin (Painlev\'e VI, V, III, Garnier system, etc) and the Sato-Segal-Wilson tau function for integrable hierarchies such as Gelfand-Dickey and Drinfeld-Sokolov.