We investigate a pilot allocation problem in wireless networks over Markovian fading channels. In wireless systems, the channel state information (CSI) is collected at the base station, in particular, this paper considers a pilot-aided channel estimation method (TDD mode). Typically, there are less available pilots than users, hence at each slot the scheduler needs to decide an allocation of pilots to users with the goal of maximizing the long-term average throughput. There is an inherent tradeoff in how the limited pilots are used: assign a pilot to a user with up-to-date CSI and good channel condition for exploitation, or assign a pilot to a user with outdated CSI for exploration. As we show, the arising pilot allocation problem is a restless bandit problem and thus its optimal solution is out of reach. In this paper, we propose an approximation based on the Lagrangian relaxation method, which provides a low-complexity Whittle index policy. We prove this policy to be asymptotically optimal in the many users regime (when the number of users in the system and the available pilots for channel sensing grow large). We evaluate the performance of Whittle's index policy in various scenarios and illustrate its remarkably good performance for small number of users, where it is not guaranteed to be optimal.