360° Image Saliency Prediction by Embedding Self-Supervised Proxy Task - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue IEEE Transactions on Broadcasting Année : 2023

360° Image Saliency Prediction by Embedding Self-Supervised Proxy Task

Résumé

The development of Metaverse industry produces many 360° images and videos. Transmitting these images or videos efficiently is the key to success of Metaverse. Since the subject’s field of view is limited in Metaverse, from the perception perspective, bit rates can be saved by focusing video encoding on salient regions. On different ways of handling 360° image projections, the existing works either consider combining local and global projections or just use only global projection for saliency prediction, which results in slow detection speed or low accuracy. In this work, we address this problem by Embedding a self-supervised Proxy task in the Saliency prediction Network, dubbed as EPSNet. The main architecture follows an autoencoder with an encoder for feature extraction and a decoder for saliency prediction. The proxy task is combined with the encoder to enforce it to learn local and global information. It is designed to find the location of a certain local projection in the global projection via self-supervised learning. A cross-attention fusion mechanism is used to fuse the global and local features for location prediction. Then, the decoder is trained based on the sole global projection. In this way, the time-consuming local-global feature fusion is placed in the training stage only. Experiments on public dataset show that our method has achieved satisfactory results in terms of inference speed and accuracy. The dataset and code are available at https://github.com/zzz0326/EPSNet.
Fichier principal
Vignette du fichier
2023_TBC_Zou_et_al.pdf (12.74 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04028523 , version 1 (16-03-2023)

Identifiants

Citer

Zizhuang Zou, Mao Ye, Shuai Li, Xue Li, Frédéric Dufaux. 360° Image Saliency Prediction by Embedding Self-Supervised Proxy Task. IEEE Transactions on Broadcasting, 2023, 69 (3), pp.704-714. ⟨10.1109/TBC.2023.3254143⟩. ⟨hal-04028523⟩
57 Consultations
17 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More