Foundation Models' Adaptation: A Case Study on Remote Sensing Imagery

Large visual-language models, such as CLIP (Contrastive Language- Image Pretraining) and SD (Stable Diffusion), demonstrate remarkable performance in general image recognition and generation tasks. Their continual learning involves two folds: enhancing their performance with more natural images as input, and adapting them to specialized image domains. Our research targets the latter, using remote sensing (RS) imagery as a use case. RS, which relies on specialized satellites, presents challenges in image annotation and suffers from data scarcity and class imbalance, especially in special spectrums. Adapting models in this domain often leads to strong biases, where features of major classes overshadow those of minor classes. To address this, we recently introduced several solutions such as debLoRA--a generic training approach compatible with various low-rank model adaptation methods (like LoRA) to produce debiased features. In this talk, we will delve into its details and its subsequent works.