A class of finite-state and discrete-time optimal control problems is introduced. The problems involve a large number of agents with independent dynamics, which interact through an aggregative term in the cost function. The problems are intractable by dynamic programming. We describe and analyze a decomposition method that only necessitates to solve at each iteration small-scale and independent optimal control problems associated with each single agent. When the number of agents is large, the convergence of the method to a nearly optimal solution is ensured, despite the absence of convexity of the problem. The procedure is based on a method called Stochastic Frank-Wolfe algorithm, designed for general nonconvex aggregative optimization problems. Numerical results are presented, for a toy model of the charging management of a battery fleet.