PixelBytes: Catching Unified Representation for Multimodal Generation - Archive ouverte HAL
Pré-Publication, Document De Travail Année : 2024

PixelBytes: Catching Unified Representation for Multimodal Generation

Fabien Furfaro
  • Fonction : Auteur

Résumé

This report introduces PixelBytes, a novel approach for unified multimodal representation learning. Inspired by existing sequence models such as Image Transformers, PixelCNN, and Mamba-Bytes, our method aims to capture diverse inputs in a cohesive representation, exploring the integration of different data types, particularly text, audio, and pixelated images (sprites). We conducted experiments on a specialized PixelBytes Pokémon dataset. Initially, we investigated various model architectures, including Recurrent Neural Networks (RNNs), State Space Models (SSMs), and Attention-based models, focusing on bidirectional processing and our convolutional PxBy embedding technique. Subsequently, we evaluated models based on data reduction strategies and the effectiveness of autoregressive learning. We specifically examined Long Short-Term Memory (LSTM) networks in both predictive and autoregressive modes for our main experiments. Our findings suggest that autoregressive models outperform predictive models in this context. By adopting a flexible approach to multimodal modeling, PixelBytes contributes to the ongoing development of foundation models capable of understanding and generating multimodal data. The complete PixelBytes project, including code, models, and datasets, is available online.
Fichier principal
Vignette du fichier
PixelBytes__Catching_Unified_Representation_for_Multimodal_Generation.pdf (822.05 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04683349 , version 1 (02-09-2024)
hal-04683349 , version 2 (14-09-2024)

Licence

Identifiants

Citer

Fabien Furfaro. PixelBytes: Catching Unified Representation for Multimodal Generation. 2024. ⟨hal-04683349v2⟩
162 Consultations
31 Téléchargements

Altmetric

Partager

More