Context.Since the 1990's, Yan and colleagues have formulated a kind of boundary integral formulation for the linear or non-linear solar force-free magnetic fields with finite energy in semi-infinite space, and developed a computational procedure by virtue of the boundary element method (BEM) to extrapolate the magnetic fields above the photosphere. Aims.In this paper, the generalized minimal residual method (GMRES) is introduced into the BEM extrapolation of the solar force-free magnetic fields, in order to efficiently solve the associated BEM system of linear equations, which was previously solved by the Gauss elimination method with full pivoting. Methods.Being a modern iterative method for non-symmetric linear systems, the GMRES method reduces the computational cost for the BEM system from to , where N is the number of unknowns in the linear system. Intensive numerical experiments are conducted on a well-known analytical model of the force-free magnetic field to reveal the convergence behaviour of the GMRES method subjected to the BEM systems. The impacts of the relevant parameters on the convergence speed are investigated in detail. Results.Taking the Krylov dimension to be 50 and the relative residual bound to be 10-6 (or 10-2), the GMRES method is at least 1000 (or 9000) times faster than the full pivoting Gauss elimination method used in the original BEM extrapolation code, when N is greater than 12 321, according to the CPU timing information measured on a common desktop computer (CPU speed 2.8 GHz; RAM 1 GB) for the model problem.