Wavelength-dependent power excursions in gaincontrolled erbium doped fiber amplifiers (EDFA) is a challenging issue in optical networks. We investigate a launch channel power control method using reinforcement learning (RL) to mitigate the power excursions of EDFA systems. A machine learning engine is developed, trained and evaluated with four different policygradient RL algorithms that are compared according to two main criteria: achieved power excursion reduction and learning time. Different scenarios are considered with 12-, 24-, 40-active channels at fixed wavelengths and with variable number of active channels (between 12 and 64) assigned randomly at different wavelengths during RL process. We show 62% power excursion reduction in the 40-channel scenario and 28% in the variable scenario, which demonstrates the promising role of online RL approach for controlling power excursion in EDFA systems.