NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
Sewon Min, Jordan Lee Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Roberts, Adam, Tom Kwiatkowski, Patrick Lewis, Yuxiang Wu, Heinrich Küttler, Liu, Linqing, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel, Sohee Yang, Minjoon Seo, Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Édouard Grave, Yamada, Ikuya, Sonse Shimaoka, Masatoshi Suzuki and 23 others
Papers open inside SciRah from the original publisher or repository; SciRah never hosts them.
Abstract
We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.