A novel implicit immersed boundary method of high accuracy and efficiency is presented for the simulation of incompressible viscous flow over complex stationary or moving solid boundaries. A boundary force is often introduced in many immersed boundary methods to mimic the presence of solid boundary, such that the overall simulation can be performed on a simple Cartesian grid. The current method inherits this idea and considers the boundary force as a Lagrange multiplier to enforce the no-slip constraint at the solid boundary, instead of applying constitutional relations for rigid bodies. Hence excessive constraint on the time step is circumvented, and the time step only depends on the discretization of fluid Navier-Stokes equations, like the CFL condition in present work. To determine the boundary force, an additional moving force equation is derived. The dimension of this derived system is proportional to the number of Lagrangian points describing the solid boundaries, which makes the method very suitable for moving boundary problems since the time for matrix update and system solving is not significant. The force coefficient matrix is made symmetric and positive definite so that the conjugate gradient method can solve the system quickly. The proposed immersed boundary method is incorporated into the fluid solver with a second-order accurate projection method as a plug-in. The overall scheme is handled under an efficient fractional step framework, namely, prediction, forcing, and projection. Various simulations are performed to validate current method, and the results compare well with previous experimental and numerical studies.