Gaussian process (GP) models have become popular for approximating and exploring non-linear systems based on scarce input/output training samples and on prior hypotheses implicitly done through prior mean and covariance functions. While it is common to make stationarity assumptions and use variance-based criteria for space exploration, in realistic test cases it is not rare that systems under study exhibit an heterogeneous behaviour depending on considered regions of the parameter space. With a class of problems in mind where high variations occur along unknown non-canonical directions, we tackle the problem of uncovering and accommodating non-stationarity in function approximation from two angles: first via a novel class of covariances (called WaMI-GP) that simultaneously generalizes kernels of Multiple Index and of tensorized warped GP models and second, by introducing derivative-based sampling criteria dedicated to the exploration of high variation regions. The novel GP class is investigated both through mathematical analysis and numerical experiments, and it is shown that proposed kernels allow encoding much expressiveness while remaining with a moderate number of parameters to be inferred. On the other hand, and independently of non-stationarity assumptions, we conduct (semi-)analytically derivations for our new variance-based infill sampling criteria relying on a change of focus from the GP to the norm of its associated gradient field. Criteria and GP models are first compared on a mechanical test case taken from nuclear safety studies conducted by IRSN. It is found on this application that some of the proposed sampling criteria including derivatives outperform usual variance-based criteria in the case of a stationary GP model, but that it is even better to use standard variance-based criteria with the proposed novel class of covariances. Comparisons are also done with the Treed Gaussian Processes (TGP) both on this application and on a three-dimensional NASA test case. In the IRSN application, WaMI-GP dominates TGP in static and sequential settings. In the NASA application, while TGP clearly dominates in the static case, for small initial designs it is outperformed by WaMI-GP in the sequential set up.