Ultrafine splittings are found in the optical absorption spectra of boron-doped diamond measured with high resolution. An analytical model of an exciton complex is developed, which permits assigning all absorption lines and sizing the interactions among the constituent charges and crystal field. We conclude that the entry of split-off holes in the acceptor-bound exciton fine structure yields two triplets separated by a spin-orbit splitting of 14.3 meV. Our findings thereby resolve a long-standing controversy [R. Sauer et al., Revised fine splitting of excitons in diamond, Phys. Rev. Lett. 84, 4172 (2000).; M. Cardona et al., Comment on “Revised fine splitting of excitons in diamond,”, Phys. Rev. Lett. 86, 3923 (2001).; R. Sauer and K. Thonke, Sauer and Thonke reply, Phys. Rev. Lett. 86, 3924 (2001).], revealing the underlying physics common in diverse semiconductors, including diamond.