In this work, we study optimistic planning methods to solve some state-constrained optimal control problems in finite horizon. While classical methods for calculating the value function are generally based on a discretization in the state space, optimistic planning algorithms have the advantage of using adaptive discretization in the control space. These approaches are therefore very suitable for control problems where the dimension of the control variable is low and allow to deal with problems where the dimension of the state space can be very high. Our algorithms also have the advantage of providing, for given computing resources, the best control strategy whose performance is as close as possible to optimality while its corresponding trajectory comply with the state constraints up to a given accuracy.