On Joint Parameterizations of Linear and Nonlinear Functionals in Neural Networks - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2023

On Joint Parameterizations of Linear and Nonlinear Functionals in Neural Networks

Résumé

The paper proposes a new class of nonlinear operators and a dual learning paradigm where optimization jointly concerns both linear convolutional weights and the parameters of these nonlinear operators. The nonlinear class proposed to perform a rich functional representation is composed by functions called rectified parametric sigmoid units. This class is constructed to benefit from the advantages of both sigmoid and rectified linear unit functions, while rejecting their respective drawbacks. Moreover, the analytic form of this new neural class involves scale, shift and shape parameters to obtain a wide range of activation shapes, including the standard rectified linear unit as a limit case. Parameters of this neural transfer class are considered as learnable for the sake of discovering the complex shapes that can contribute to solving machine learning issues. Performance achieved by the joint learning of convolutional and rectified parametric sigmoid learnable parameters are shown to be outstanding in both shallow and deep learning frameworks. This class opens new prospects with respect to machine learning in the sense that main learnable parameters are attached not only to linear transformations, but also to a wide range of nonlinear operators.
Fichier principal
Vignette du fichier
DeepSigmoidCNN_V4ArXiv/DeepSigmoidCNNArXiv.pdf (2.83 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03115681 , version 1 (22-01-2021)
hal-03115681 , version 2 (02-02-2021)
hal-03115681 , version 3 (12-01-2023)

Identifiants

Citer

Abdourrahmane Mahamane Atto, Sylvie Galichet, Dominique Pastor, Nicolas Méger. On Joint Parameterizations of Linear and Nonlinear Functionals in Neural Networks. 2023. ⟨hal-03115681v2⟩
152 Consultations
85 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More