The definition of a metric between time series is inherent to several data analysis and mining tasks, including clustering, classification or forecasting. Time series data present naturally several modalities covering their amplitude, behavior or frequential spectrum, that may be expressed with varying delays and at multiple temporal scales—exhibited globally or locally. Combining several modalities at multiple temporal scales to learn a holistic metric is a key challenge for many real temporal data applications. This paper proposes a Multi-modal and Multi-scale Temporal Metric Learning (m 2 tml) approach for a robust time series nearest neighbors classification. The solution lies in embedding time series into a dissimilarity space where a pairwise svm is used to learn both linear and non linear combined metric. A sparse and interpretable variant of the solution shows the ability of the learned temporal metric to localize accurately discriminative modalities as well as their temporal scales. A wide range of 30 public and challenging datasets, encompassing images , traces and ecg data, are used to show the efficiency and the potential of m 2 tml for an effective time series nearest neighbors classification.