Delving into Pre-training for Domain Transfer: A Broad Study of Pre-training for Domain Generalization and Domain Adaptation

As deep learning models suffer from domain shifts, domain transfer methods have been developed to learn robust and reliable feature representations on unseen domains. Existing domain transfer methods, such as domain adaptation and domain generalization, focused on developing new adaptation or alignment algorithms, typically utilizing outdated ResNet backbones pre-trained on ImageNet-1K. However, the impact of recent pre-training approaches on domain transfer has not been thoroughly investigated. In this work, we provide a broad study and in-depth analysis of pre-training for domain adaptation and generalization from four distinct perspectives; network architectures, sizes, pre-training objectives, and pre-training datasets. Our extensive experiments cover a variety of domain transfer settings, including domain generalization, unsupervised domain adaptation, source free domain adaptation, and universal domain adaptation. Our study reveals two key findings: (1) state-of-the-art pre-training has a greater impact on performance than advanced generalization or adaptation techniques, (2) domain adaptation baselines tend to overfit to older pre-training backbones, indicating that top-performing methods under previous settings may no longer be optimal with modern pre-training, and (3) these trends are also observed in other tasks, such as object detection and semantic segmentation. Furthermore, we investigate what makes pre-training effective for domain transfer. Interestingly, our findings suggest that the performance gains are largely due to the presence of a significantly higher number of classes in recent pre-training datasets (e.g., ImageNet-22K) that closely resemble those in downstream tasks, rather than solely the result of large-scale data. In addition, we examine potential train/test contamination between web-scale pre-training datasets and downstream benchmarks and find that such data leakage has only a negligible impact on evaluation. We hope this work highlights the importance of pre-training for domain transfer and offers valuable insights for future domain transfer research.