Reinforcement learning is a commonly used technique in the field of traffic signal control. By iteratively testing actions given the current network state, a traffic signal agent gradually learns to optimally control traffic lights given the traffic situation at hand. While they usually outperform traditional control systems in the literature, these methods have to first go through an exploration phase where they test different actions in a trial-and-error fashion. The nature of reinforcement learning methods hence causes unstable performances and high computational costs. In order to limit the costs linked to this exploration phase, we propose a bootstrapping method that reduces the computation time needed for learning-based traffic control methods to reach acceptable performance levels. Our method models lanes of the road network as queues, and derives results on the average service time of vehicles at the intersection level in order to estimate the agent's policy. The performance of our bootstrapping method is then compared to a more traditional Q-Learning method using the SUMO simulator. Simulation results show that bootstrapping alleviates both of these issues by immediately reaching acceptable performance levels by quickly training the agent without any direct interaction with the simulation environment.