We search for passive galaxies at z > 3 in the GOODS-South field, using different techniques based on photometric data, and paying attention to develop methods that are sensitive to objects that have become passive shortly before the epoch of observation. We use CANDELS HST catalogues, ultra-deep Ks data and new IRAC photometry, performing spectral energy distribution fitting using models with abruptly quenched star formation histories. We then single out galaxies which are best fitted by a passively evolving model, and having only low probability (<5 per cent) star-forming solutions. We verify the effects of including nebular lines emission, and we consider possible solutions at different redshifts. The number of selected sources dramatically depends on the models used in the spectral energy distribution (SED) fitting. Without including emission lines and with photometric redshifts fixed at the CANDELS estimate, we single out 30 candidates; the inclusion of nebular lines emission reduces the sample to 10 objects; allowing for solutions at different redshifts, only two galaxies survive as robust candidates. Most of the candidates are not far-infrared emitters, corroborating their association with passive galaxies. Our results translate into an upper limit in the number density of ∼0.173 arcmin 2 above the detection limit. However, we conclude that the selection of passive galaxies at z > 3 is still subject to significant uncertainties, being sensitive to assumptions in the SED modelling adopted and to the relatively low S/N of the objects. By means of dedicated simulations, we show that JWST will greatly enhance the accuracy, allowing for a much more robust classification.