Both the extent and accuracy of force application in atomic force microscope (AFM) nanomanipulation are significantly limited by the nonlinearity of the commonly used optical lever with a nonlinear position-sensitive detector (PSD). In order to compensate the nonlinearity of the optical lever, a nonlinear calibration method is presented. This method applies the nonlinear curve fit to a full-range position-voltage response of the photodiode, obtaining a continuous function of its voltage-related sensitivity. Thus, Interaction forces can be defined as integrals of this sensitivity function between any two responses of photodiode voltage outputs, instead of rough transformation with a single conversion factor. The lateral position-voltage response of the photodiode, a universally acknowledged puzzle, was directly characterized by an accurately calibrated force sensor composed of a tippless piezoresistive force sensor, regardless of any knowledge of the cantilevers and laser measuring system. Experiments using a rectangular cantilever (normal force constant 0.24 N/m) demonstrated that the proposed nonlinear calibration method restrained the sensitivity error of normal position-voltage responses to 3.6% and extended the force application range.