Conic Linear Units: Improved Model Fusion and Rotational-Symmetric Generative Model - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2024

Conic Linear Units: Improved Model Fusion and Rotational-Symmetric Generative Model

Résumé

We introduce Conic Linear Unit (CoLU), a natural generalization of commonly used activation functions in neural networks. The common pointwise ReLU activation is a projection onto the positive cone and is permutation symmetric. We propose a nonlinearity that goes beyond this symmetry: CoLU is a skew projection onto a hypercone towards the cone's axis. Due to the nature of this projection, CoLU enforces symmetry in a neural network with width C from the finite-order permutation group S(C) to the infinite-order rotation/reflection group O(C-1), thus producing deep features that are motivated by the HSV color representation. Recent results on merging independent neural networks via permutation modulus can be relaxed and generalized to soft alignment modulo an optimal transport plan, which is useful in aligning models of different widths. CoLU aims to further alleviate the apparent deficiency of soft alignment. Our simulation indicates that CoLU outperforms existing generative models including Autoencoder and Latent Diffusion Model on small or large-scale image datasets. Additionally, CoLU does not increase the number of parameters and requires negligible additional computation overhead. The CoLU concept is quite general and can be plugged into various neural network architectures. Ablation studies on extensions to soft projections, general Lp cones, and the non-convex double-cone cases are briefly discussed.
Fichier principal
Vignette du fichier
colu_final.pdf (12.78 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Licence : CC BY - Paternité

Dates et versions

hal-04383027 , version 1 (16-01-2024)

Licence

Paternité

Identifiants

  • HAL Id : hal-04383027 , version 1

Citer

Changqing Fu, Laurent D. Cohen. Conic Linear Units: Improved Model Fusion and Rotational-Symmetric Generative Model. International Conference on Computer Vision Theory and Applications, Feb 2024, Rome, Italy. ⟨hal-04383027⟩
116 Consultations
24 Téléchargements

Partager

Gmail Facebook X LinkedIn More