Edge-Conditioned Feature Transform Network for Hyperspectral and Multispectral Image Fusion - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue IEEE Transactions on Geoscience and Remote Sensing Année : 2022

Edge-Conditioned Feature Transform Network for Hyperspectral and Multispectral Image Fusion

Yuxuan Zheng
Jiaojiao Li
Connectez-vous pour contacter l'auteur
Yunsong Li
Connectez-vous pour contacter l'auteur
Jie Guo
  • Fonction : Auteur
Xianyun Wu
Yanzi Shi

Résumé

Despite recent advances achieved by deep learning techniques in the fusion of low-spatial-resolution hyperspectral image (LR-HSI) and high-spatial-resolution multispectral image (HR-MSI), it remains a challenge to reconstruct the high-spatial-resolution HSI (HR-HSI) with more accurate spatial details and less spectral distortions, since the low-level structure information such as sharp edges tends to be weakened or lost as the network depth grows. To tackle this issue, we creatively propose an edge-conditioned feature transform network (EC-FTN) in this article, which is mainly composed of three parts, namely, feature extraction network (FEN), feature fusion and transformation network (FFTN), and image reconstruction network (IRN). First, two computationally efficient FENs with 3-D convolutions and reshaping layers are employed to extract the joint spectral-spatial features of input images. Then, the FFTN conditioned on the edge map prior can fuse and transform the features adaptively, in which a fusion node and several cascaded feature modulation modules (FMMs) equipped with feature-wise modulation layers are constructed. Specifically, the edge map is generated via transfer learning, i.e., by applying the Sobel operator to feature maps of the red-green-blue (RGB) version of HR-MSI resulting from the pretrained VGG16 model without extra training. Finally, the desired HR-HSI is recovered from the transformed features through IRN. Furthermore, we elaborately design a weighted combinatorial loss function consisting of mean absolute error, image gradient difference, and spectral angle terms to guide the training. Experiments on both ground-based and remotely sensed datasets demonstrate that our EC-FTN outperforms state-of-the-art methods in visual and quantitive evaluations, as well as in fine details reconstruction.
Fichier non déposé

Dates et versions

hal-03932750 , version 1 (10-01-2023)

Identifiants

Citer

Yuxuan Zheng, Jiaojiao Li, Yunsong Li, Jie Guo, Xianyun Wu, et al.. Edge-Conditioned Feature Transform Network for Hyperspectral and Multispectral Image Fusion. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60, pp.5513315. ⟨10.1109/TGRS.2021.3108122⟩. ⟨hal-03932750⟩
20 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More