Reverse-engineering of high-dimensional dynamical systems from time-course data still remains a challenging and important problem in knowledge discovery. For this learning task, a number of approaches primarily based on sparse linear models or Granger causality concepts have been proposed in the literature. However, when a system exhibits nonlinear dynamics, there does not exist a systematic approach that takes into account the nature of the underlying system. In this work, we introduce a novel family of vector autoregressive models based on different operator-valued kernels to identify the dynamical system and retrieve the target network that characterizes the interactions of its components. Assuming a sparse underlying structure, a key challenge, also present in the linear case, is to control the model's sparsity. This is achieved through the joint learning of the structure of the kernel and the basis vectors. To solve this learning task, we propose an alternating optimization algorithm based on proximal gradient procedures that learns both the structure of the kernel and the basis vectors. Results on the DREAM3 competition gene regulatory benchmark networks of sizes 10 and 100 show the new model outperforms existing methods. Another application of the model on climate data identifies interesting and interpretable interactions between natural and human activity factors, thus confirming the ability of the learning scheme to retrieve dependencies between state-variables.