DARTS with Degeneracy Correction
Résumé
The neural architecture search (NAS) is characterized by a wide search space and a time consuming objective function. Many papers have dealt with the reduction of the cost of the objective function assessment. Among them, there is DARTS paper [1] that proposes to transform the original discrete problem into a continuous one. This paper builds an overparameterized network called hypernetwork and weights its edges by continuous coefficients. This approach allows to considerably reduce the computational cost, but the quality of the obtained architectures is highly variable. We propose to reduce this variability by introducing a convex depth regularization. We also add a heuristic that controls the number of unweighted operations. The goal is to correct the short term bias, introduced by the hypergradient approximation. Finally, we will show the efficiency of these proposals by starting again the work developed in Dots paper [2].