PaLI-X: On Scaling up a Multilingual Vision and Language Model
Chen, Xia, Josip Djolongaa, Piotr Padlewskia, Basil Mustafaa, Soravit Changpinyoa, Jialin Wua, Carlos Riquelmea, Sebastian Goodmana, Xiao Wanga, Yi Taya, Siamak Shakeria, Mostafa Dehghania, Daniel Salza, Mario Lučića, Michael Tschannena, Arsha Nagrania, Hexiang Hua, Joshi, Mandara, Bo Panga, Ceslee Montgomerya, Pietrzyk, Paulinaa, Marvin Rittera, AJ Piergiovannia, Matthias Minderera, Filip Pavetica, Waters, Austina, Gang Lia, Ibrahim Alabdulmohsina, Lucas Beyera, Julien Amelota and 13 others
- aGoogle Research
Papers open inside SciRah from the original publisher or repository; SciRah never hosts them.
Abstract
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-shot (in-context) learning, as well as object detection, video question answering, and video captioning. PaLI-X advances the state-of-the-art on most vision-and-language benchmarks considered (25+ of them). Finally, we observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.
TL;DRPaLI-X, a multilingual vision and language model, advances the state-of-the-art on most vision-and-language benchmarks considered and observes emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix. (Semantic Scholar)