An Optimal Transport Kernel for Feature Aggregation and its Relationship to Attention - Archive ouverte HAL
Pré-Publication, Document De Travail Année : 2020

An Optimal Transport Kernel for Feature Aggregation and its Relationship to Attention

Résumé

We introduce a kernel for sets of features based on an optimal transport distance, along with an explicit embedding function. Our approach addresses the problem of feature aggregation, or pooling, for sets that exhibit long-range dependencies between their members. More precisely, our embedding aggregates the features of a given set according to the transport plan between the set and a reference shared across the data set. Unlike traditional hand-crafted kernels, our embedding can be optimized for a specific task or data set. It also has a natural connection to attention mechanisms in neural networks, which are commonly used to deal with sets, yet requires less data. Our embedding is particularly suited for biological sequence classification tasks and shows promising results for natural language sequences. We provide an implementation of our embedding that can be used alone or as a module in larger learning models. Our code is freely available at https://github.com/claying/OTK.
Fichier principal
Vignette du fichier
main_otk.pdf (1.5 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02883436 , version 1 (29-06-2020)
hal-02883436 , version 2 (05-10-2020)
hal-02883436 , version 3 (09-02-2021)

Identifiants

  • HAL Id : hal-02883436 , version 1

Citer

Grégoire Mialon, Dexiong Chen, Alexandre d'Aspremont, Julien Mairal. An Optimal Transport Kernel for Feature Aggregation and its Relationship to Attention. 2020. ⟨hal-02883436v1⟩
6536 Consultations
790 Téléchargements

Partager

More