In this paper, a novel method for computing the nonlinear dynamics of highly flexible slender structures in three dimensions (3D) is proposed. It is the extension to 3D of a previous work restricted to inplane (2D) deformations. It is based on the geometrically exact beam model, which is discretized with a finite-element method and solved entirely in the frequency domain with a harmonic balance method (HBM) coupled to an asymptotic numerical method (ANM) for continuation of periodic solutions. An important consideration is the parametrization of the rotations of the beam’s cross sections, much more demanding than in the 2D case. Here, the rotations are parametrized with quaternions, with the advantage of leading naturally to polynomial nonlinearities in the model, well suited for applying the ANM. Because of the HBM–ANM framework, this numerical strategy is capable of computing both the frequency response of the structure under periodic oscillations and its nonlinear modes (namely its backbone curves and deformed shapes in free conservative oscillations). To illustrate and validate this strategy, it is used to solve two 3D deformations test cases of the literature: a cantilever beam and a clamped–clamped beam subjected to one-to-one (1:1) internal resonance between two companion bending modes in the case of a nearly square cross section.