In this paper, the instability of a fluid inside a precessing cylinder is addressed theoretically and experimentally. The precessional motion forces Kelvin modes in the cylinder, which can become resonant for given precessional frequencies and cylinder aspect ratios. When the Reynolds number is large enough, these forced resonant Kelvin modes eventually become unstable. A linear stability analysis based on a triadic resonance between a forced Kelvin mode and two additional free Kelvin modes is carried out. This analysis allows us to predict the spatial structure of the instability and its threshold. These predictions are compared to the vorticity field measured by particle image velocimetry with an excellent agreement. When the Reynolds number is further increased, nonlinear effects appear. A weakly nonlinear theory is developed semi-empirically by introducing a geostrophic mode, which is triggered by the nonlinear interaction of a free Kelvin mode with itself in the presence of viscosity. Amplitude equations are obtained coupling the forced Kelvin mode, the two free Kelvin modes and the geostrophic mode. They show that the instability saturates to a fixed point just above threshold. Increasing the Reynolds number leads to a transition from a steady saturated regime to an intermittent flow in good agreement with experiments. Surprisingly, this weakly nonlinear model still gives a correct estimate of the mean flow inside the cylinder even far from the threshold when the flow is turbulent.