conference paper

On Scaling Up a Multilingual Vision and Language Model

Xi Chen^a, Josip Djolonga^a, Piotr Padlewski^a, Basil Mustafa^a, Soravit Changpinyo^a, Jialin Wu^a, Carlos Riquelme^a, Sebastian Goodman^a, Xiao Wang^a, Yi Tay^a, Siamak Shakeri^a, Mostafa Dehghani^a, Daniel Salz^a, Mario Lučić^a, Michael Tschannen^a, Arsha Nagrani^a, Hexiang Hu^a, Mandar Joshi^a, Bo Pang^a, Ceslee Montgomery^a, Paulina Pietrzyk^a, Marvin Ritter^a, AJ Piergiovanni^a, Matthias Minderer^a, Filip Pavetic^a, Austin Waters^a, Gang Li^a, Ibrahim Alabdulmohsin^a, Lucas Beyer^a, Julien Amelot^a and 13 others

^aGoogle

IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings Institute of Electrical and Electronics Engineers2024-06-16 52 citations86 references

Looking for the full paper…

Export

Papers open inside SciRah from the original publisher or repository; SciRah never hosts them.

Abstract

We explore the boundaries of scaling up a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-shot (in-context) learning, as well as object detection, video question answering, and video captioning. Our model advances the state-of-the-art on most vision-and-language benchmarks considered (20+ of them). Finally, we observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.

Plain-language summary

AI illustration of the topic of “On Scaling Up a Multilingual Vision and Language Model”

On Scaling Up a Multilingual Vision and Language Model

Abstract

Plain-language summary

Discussion