With the increasingly pervasive use of advanced machine learning models comes the need to explain their predictions. The SHAP framework, based on Shapley values, provides explanations to highlight which features could be important for a given prediction. However, its use is hampered by its computational cost, especially in its model-agnostic formulation. Model-specific algorithms offer a restricted solution to this problem, whereas alternative approximation strategies can maintain the model-agnostic property. Here we propose TopShap as an agnostic algorithm that searches the k most important features by interleaving pruning of candidates and refinement of the approximate SHAP values. TopShap is built on three insights: (i) it performs an iterative approximation taking advantage of a previously developed sampling strategy, (ii) it uses confidence interval bounds around approximate SHAP values to determine on-the-fly which features can no longer be part of the top-k, (iii) it stops when these interval bounds are stable. Evaluating TopShap on publicly available datasets shows it performs an effective pruning of the feature search space and leads to an important reduction of the execution cost when compared to the other agnostic approaches. We also apply TopShap to a use case in biology and show that top-k search is meaningful in this context.