Wind farms are subject to the so-called "wake effect", where upstream turbines facing the wind create sub-optimal wind conditions for turbines located downstream. One strategy to address this issue is to use yaw actuators to misalign the wind turbines with regard to the incoming wind direction, thus deflecting wakes away from downstream turbines. Tractable models for yaw optimization are however subject to inaccuracies, ignore wake dynamics and lack adaptability. This incentivizes the use of model-free methods. In this paper, we propose a delay-aware decentralized Q-learning algorithm for yaw control on wind farms. We introduce a strategy to handle delayed cost collection, and show that our method significantly increases power production in simulations with realistic wake dynamics. We validate our results for two farm layouts on midfidelity wind farm simulator FAST.Farm.