The paper explores in detail the use of dynamic adaptation gain/learning rate (DAG) for improving the performance of gradient type adaptation/learning algorithms. The DAG is an ARMA (poles-zeros) filter embedded in the gradient type adaptation/learning algorithms and generalizes the various improved gradient algorithms available in the literature. After introducing the DAG algorithm and its relation with other algorithms, its design is developed. Strictly Positive Real (SPR) conditions play an important role in the design of the DAG. Then the stability issues for adaptive/learning systems using a DAG are discussed for large and low values of the adaptation gains/learning rate. The potential of the DAG is then illustrated by experimental results obtained on a relevant adaptive active noise control system.