preprint Open access

GLU Variants Improve Transformer

^aGoogle
✉Corresponding author

arXiv (Cornell University)Cornell University2020-02-12 302 citations11 references

Looking for the full paper…

Export

Papers open inside SciRah from the original publisher or repository; SciRah never hosts them.

Abstract

Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer (arXiv:1706.03762) sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations.

Plain-language summary

AI illustration of the topic of “GLU Variants Improve Transformer”

GLU Variants Improve Transformer

Abstract

Plain-language summary

Discussion