Exploring the Effect of Multi-Step Ascent in Sharpness-Aware Minimization
Hoki Kima, Jinseong Parkb, Woo-Jin Leec, Jaewook Leed, Yujin Choie
- aDepartment of Industrial Security, Chung-Ang University, Seoul, South Korea
- bKorea Institute for Advanced Study, Seoul, South Korea
- cGraduate School of Computer Science and Artificial Intelligence, Dongguk University, Seoul, South Korea
- dDepartment of Industrial Engineering, Seoul National University, Seoul, South Korea
- eDepartment of Industrial Engineering, Ulsan National Institute of Science and Technology, Ulsan, South Korea
Papers open inside SciRah from the original publisher or repository; SciRah never hosts them.
Abstract
Recently, Sharpness-Aware Minimization (SAM) has shown state-of-the-art performance by seeking flat minima. To minimize the maximum loss within a neighborhood in the parameter space, SAM uses an ascent step, which perturbs the weights along the direction of gradient ascent with a given radius. While single-step or multi-step can be taken during ascent steps, previous studies have shown that multi-step ascent SAM rarely improves generalization performance. However, this phenomenon is particularly interesting because the multi-step ascent is expected to provide a better approximation of the maximum neighborhood loss. Therefore, in this paper, we analyze the effect of the number of ascent steps and investigate the difference between both single-step ascent SAM and multi-step ascent SAM. We identify the effect of the number of ascents on SAM optimization and reveal that single-step ascent SAM and multi-step ascent SAM exhibit distinct loss landscapes. Based on these observations, we finally suggest a simple modification that can mitigate the suboptimal generalization of multi-step ascent SAM.
TL;DRIt is revealed that single-step ascent SAM and multi-step ascent SAM exhibit distinct loss landscapes and a simple modification is suggested that can mitigate the suboptimal generalization of multi-step ascent SAM. (Semantic Scholar)