Residual Prompt Tuning: improving prompt tuning with residual reparameterization

Prompt tuning is one of the successful approaches for parameter-efficient tuning of pretrained language models.Despite being arguably the most parameter-efficient (tuned soft prompts constitute < 0.1% of total parameters), it typically performs worse than other efficient tuning methods and is quite sensitive to hyper-parameters.In this work, we introduce RESIDUAL PROMPT TUNING -a simple and efficient method that significantly improves the performance and stability of prompt tuning.We propose to reparameterize soft prompt embeddings using a shallow network with a residual connection.Our experiments show that RESIDUAL PROMPT TUNING significantly outperforms prompt tuning on SuperGLUE benchmark across T5-Large, T5-Base and BERT-Base models.Notably, our method reaches +7 points improvement over prompt tuning with T5-Base and allows to reduce the prompt length by ×10 without hurting performance.In addition, we show that our approach is robust to the choice of learning rate and prompt initialization, and is effective in few-shot settings.1