We introduce horizontal holonomy groups, which are groups defined using parallel transport only along curves tangent to a given subbundle $D$ of the tangent bundle. We provide explicit means of computing these holonomy groups by deriving analogues of Ambrose-Singer's and Ozeki's theorems. We then give necessary and sufficient conditions in terms of the horizontal holonomy groups for existence of solutions of two problems on foliated manifolds: determining when a foliation can be either (a) totally geodesic or (b) endowed with a principal bundle structure. The subbundle $D$ plays the role of an orthogonal complement to the leaves of the foliation in case (a) and of a principal connection in case (b).