The Impact of Action in Visual Representation Learning - Archive ouverte HAL
Conference Papers Year : 2022

The Impact of Action in Visual Representation Learning

Abstract

Sensori-motor theories, inspired by work in neuroscience, psychology and cognitive science, claim that actions, through learning and mastering of a predictive model, are a key element in the perception of the environment. On the computational side, in the domains of representation learning and reinforcement learning, models are increasingly using self-supervised pretext tasks, such as predictive or contrastive ones, in order to increase the performance on their main task. These pretext tasks are actionrelated even if the action itself is usually not used in the model. In this paper, we propose to study the influence of considering action in the learning of visual representations in deep neural network models, an aspect which is often underestimated w.r.t. sensorimotor theories. More precisely, we quantify two independent factors: 1-whether or not to use the action during the learning of visual characteristics, and 2-whether or not to integrate the action in the representations of the current images. Other aspects will be kept as simple and comparable as possible, that is why we will not consider any specific action policies and combine simple architectures (VAE and LSTM), while using datasets derived from MNIST. In this context, our results show that explicitly including action in the learning process and in the representations improves the performance of the model, which opens interesting perspectives to improve state-of-the-art models of representation learning.
Fichier principal
Vignette du fichier
ICDL_v1.pdf (1.46 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03815546 , version 1 (14-10-2022)

Identifiers

Cite

Alexandre Devillers, Valentin Chaffraix, Frédéric Armetta, Stefan Duffner, Mathieu Lefort. The Impact of Action in Visual Representation Learning. International Conference on Development and Learning (ICDL 2022), Sep 2022, London, United Kingdom. ⟨10.1109/ICDL53763.2022.9962210⟩. ⟨hal-03815546⟩
57 View
155 Download

Altmetric

Share

More