We consider the linear and quadratic higher order terms associated to the response of the statistical properties of a dynamical system to suitable small perturbations. These terms are related to the first and second derivative of the stationary measure with respect to the change of some parameters, expressing how the statistical properties of the system varies under the perturbation. We show a general framework in which one can obtain rigorous convergence and formulas for these two terms. The framework is flexible enough to be applied both to deterministic and random systems. We give examples of such an application computing linear and quadratic response for Arnold maps with additive noise and deterministic expanding maps.