We investigate the isometry groups of the left-invariant Riemannian and sub-Riemannian structures on simply connected three-dimensional Lie groups. More specifically, we determine the isometry group for each normalized structure and hence characterize for exactly which structures (and groups) the isotropy subgroup of the identity is contained in the group of automorphisms of the Lie group. It turns out (in both the Riemannian and sub-Riemannian cases) that for most structures any isometry is the composition of a left translation and a Lie group automorphism.