A general class of large-scale, nonconvex, and non-smooth optimization problems is introduced. It has the form of a multi-agent problem, where the agents interact through an aggregative term. A convex relaxation of the problem is provided together with an estimate of the relaxation gap. A numerical method, called stochastic Frank-Wolfe algorithm, is presented. The method allows to find approximate solutions of the original problem in a decomposed fashion. The convergence of the method is guaranteed from a theoretical point of view. An aggregative deterministic optimal control problem is formulated, with discrete state-space and discrete time. It is shown that the stochastic Frank-Wolfe algorithm can be applied to the optimal control problem; in particular, it amounts to solve at each iteration a series of small-scale optimal control problems, corresponding to each agent. These sub-problems are solved by dynamic programming. Numerical results are presented, for a toy model of the charging management of a battery fleet.