This article introduces a distributed convex optimization algorithm in a constrained multi-agent system composed by a large number of nodes. We focus on the case where each agent seeks to optimize its own local parameter under few coupling equality and inequality constraints. The objective function is of the power flow type and can be decoupled as a sum of elementary functions, each of which assumed (imperfectly) known by only one node. Under these assumptions, a cost-efficient decentralized iterative solution based on Lagrangian duality is derived, which is provably converging. This new approach alleviates several limitations of algorithms proposed in the stochastic optimization literature. Applications are proposed to decentralized power flow optimization in smart grids.