Wireless underground sensor networks (WUSNs) consist of sensors that are buried in and communicate through soil medium, while the channel quality of WUSNs is greatly impacted by the underground environment, such as soil moisture and composition. Due to the precipitation and harsh weather, the underground environments change frequently, which make wireless communication in WUSNs much complicated than that in terrestrial over-the-air wireless sensor networks. To achieve reliable and energy-efficient data gathering in dynamic WUSNs, this article proposes an optimal transmission policy, where path loss of sensory data transmission, energy constraint, and network load balancing are the factors to be considered. Specifically, we capture the effect of underground environments on wireless communications, and evaluate path probability, energy consumption, and load balancing factor with respect to reliability and efficiency of transmission paths. The transmission topology can be reduced to a multi-objective and multi-constrained optimization problem and solved through an improved maximum flow minimum cost algorithm. By using reinforcement learning, we derive an adaptive transmission policy for underground sensors to efficiently use their energy and avoid transmitting sensory data in unreliable paths under a dynamic environment. Through simulations and comparison upon publicly available real data, our technique achieves more reliable wireless communication with significant reduction of packet loss, and enables more energy-efficient data gathering than other techniques, especially when soil moisture varies frequently.