The rates of quantum cryptographic protocols are usually expressed in terms of a conditional entropy minimized over a certain set of quantum states. In particular, in the device-independent setting, the minimization is over all the quantum states jointly held by the adversary and the parties that are consistent with the statistics that are seen by the parties. Here, we introduce a method to approximate such entropic quantities. Applied to the setting of device-independent randomness generation and quantum key distribution, we obtain improvements on protocol rates in various settings. In particular, we find new upper bounds on the minimal global detection efficiency required to perform device-independent quantum key distribution without additional preprocessing. Furthermore, we show that our construction can be readily combined with the entropy accumulation theorem in order to establish full finite-key security proofs for these protocols. In order to achieve this we introduce the family of iterated mean quantum R\'enyi divergences with parameters $\alpha_k = 1+\frac{1}{2^{k}-1}$ for positive integers $k$. We then show that the corresponding conditional entropies admit a particularly nice form which, in the context of device-independent optimization, can be relaxed to a semidefinite programming problem using the Navascu\'es-Pironio-Ac\'in hierarchy.