We theoretically study the heat conduction in a harmonic chain with a stochastic force field, in contact with two Langevin thermal baths at different temperatures. In particular, we investigate the interplay between the thermal baths properties and the size of the system. To this aim, we introduce a stochastic force field, which is simple enough to be energy-conserving for each particle of the chain, but sufficiently aleatory to induce the ballistic to diffusive transition in the conductive behavior of the chain. When this stochastic force field is absent, we observe a ballistic behavior strongly dependent on the characteristic collision frequency of the thermal baths for any size of the system. On the other hand, when the stochastic force field is activated, the diffusive behavior is established and the effect of the thermal baths is removed in the thermodynamic limit.