Simple and Efficient ways to Improve REALM
Vidhisha Balachandrana, Ashish Vaswanib, Yulia Tsvetkovc, Niki Parmarb
- aLanguage Technologies Institute , Carnegie Mellon University
- bGoogle Research
- cPaul G. Allen School of Computer Science & Engineering , University of Washington
Papers open inside SciRah from the original publisher or repository; SciRah never hosts them.
Abstract
Dense retrieval has been shown to be effective for Open Domain Question Answering, surpassing sparse retrieval methods like BM25.One such model, REALM, (Guu et al., 2020) is an end-to-end dense retrieval system that uses MLM based pretraining for improved downstream QA performance.However, the current REALM setup uses limited resources and is not comparable in scale to more recent systems, contributing to its lower performance.Additionally, it relies on noisy supervision for retrieval during fine-tuning.We propose REALM++, where we improve upon the training and inference setups and introduce better supervision signal for improving performance, without any architectural changes.REALM++ achieves ∼5.5% absolute accuracy gains over the baseline while being faster to train.It also matches the performance of large models which have 3x more parameters demonstrating the efficiency of our setup.
TL;DRThis work proposes REALM++, where it improves upon the training and inference setups and introduces better supervision signal for improving performance, without any architectural changes, and achieves ~5.5% absolute accuracy gains over the baseline while being faster to train. (Semantic Scholar)