preprint Open access

Implicit Bias of Gradient Descent on Linear Convolutional Networks

Suriya Gunasekar^{a b}, Jason D. Lee^{c d}, Daniel Soudry^{e f}, Nathan Srebro^{a b}

^aTTI at Chicago , USA
^bToyota Technological Institute at Chicago
^cUSC Los Angeles , USA
^dUniversity of Southern California
^eTechnion , Israel
^fTechnion-Israel Institute of Technology

arXiv (Cornell University)Cornell University2018-06-01 75 citations28 references

Looking for the full paper…

Export

Papers open inside SciRah from the original publisher or repository; SciRah never hosts them.

Abstract

We show that gradient descent on full-width linear convolutional networks of depth $L$ converges to a linear predictor related to the $\ell_{2/L}$ bridge penalty in the frequency domain. This is in contrast to linearly fully connected networks, where gradient descent converges to the hard margin linear support vector machine solution, regardless of depth.

Plain-language summary

AI illustration of the topic of “Implicit Bias of Gradient Descent on Linear Convolutional Networks”

Implicit Bias of Gradient Descent on Linear Convolutional Networks

Abstract

Plain-language summary

Discussion