【CVPR 2025】MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders
Jiajun Cao, Yuan Zhang, Tao Huang, Ming Lu, Qizhe Zhang, Ruichuan An, Ningning MA, Shanghang Zhang

Visual encoders are fundamental components in vision-language models (VLMs), each showcasing unique strengths derived from various pre-trained visual foundation models. To leverage the various capabilities of these encoders, recent studies incorporate multiple encoders within a single VLM, leading to a considerable increase in computational cost. In this paper, we present Mixture-of-Visual-Encoder Knowledge Distillation (MoVE-KD), a novel framework that distills the unique proficiencies of multiple vision encoders into a single, efficient encoder model. Specifically, to mitigate conflicts and retain the unique characteristics of each teacher encoder, we employ low-rank adaptation (LoRA) and mixture-of-experts (MoEs) to selectively activate specialized knowledge based on input features, enhancing both adaptability and efficiency. To regularize the KD process and enhance performance, we propose an attention-based distillation strategy that adaptively weighs the different visual encoders and emphasizes valuable visual tokens, reducing the burden of replicating comprehensive but distinct features from multiple teachers.

• [2025/2/26] 🔥 The code of MoVE-KD is released.

• [2025/2/26] 🔥 MoVE-KD is accepted by CVPR 2025.

Usage and License Notices: This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses, including but not limited to the OpenAI Terms of Use for the dataset and the specific licenses for base language models for checkpoints trained using the dataset (e.g. Llama community license for LLaMA-2 and Vicuna-v1.5). This project does not impose any additional constraints beyond those stipulated in the original licenses. Furthermore, users are reminded to ensure that their use of the dataset and checkpoints is in compliance with all applicable laws and regulations.

• Install
• MoVE-KD Weights
• Train
• Evaluation

If you are not using Linux, do NOT proceed, see instructions for macOS and Windows.

1. Clone this repository and navigate to LLaVA folder

git clone https://github.com/hey-cjj/MoVE-KD.git
cd MoVE-KD

2. Install Package

conda create -n move-kd python=3.10 -y
conda activate move-kd
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

3. Install additional packages for training cases

pip install -e ".[train]"
pip install flash-attn --no-build-isolation

Our training procedure is consistent with LLaVA, consisting of two stages: pre-training and fine-tuning. In the pre-training stage, we tune the parameter weights of the student encoder’s MoLE, encoder adapters, and projector, while freezing all other parameter weights. In the fine-tuning stage, all parameter weights are updated except those of the teacher encoders.

Training script with DeepSpeed ZeRO-2: pretrain-v1.0.sh and pretrain-v1.1.sh.

Training script with DeepSpeed ZeRO-2: finetune-v1.0.sh and finetune-v1.1.sh.

See Evaluation.md.

If you find MoVE-KD useful for your research and applications, please cite using this BibTeX:

@InProceedings{Cao_2025_CVPR,
    author    = {Cao, Jiajun and Zhang, Yuan and Huang, Tao and Lu, Ming and Zhang, Qizhe and An, Ruichuan and Ma, Ningning and Zhang, Shanghang},
    title     = {MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {19846-19856}
}

• Our code is built on LLaVA and EAGLE