video-SALMONN-2
video-SALMONN 2 is a powerful audio-visual large language model (LLM) that generates high-quality audio-visual video captions, which is developed by the Departm
What is video-SALMONN-2?
video-SALMONN 2 is a powerful audio-visual large language model (LLM) that generates high-quality audio-visual video captions, which is developed by the Department of Electronic Engineering at Tsinghua University and ByteDance.
Overview of video-SALMONN-2
What it is
video-SALMONN 2 is an audio-visual large language model (LLM) developed by the Department of Electronic Engineering at Tsinghua University and ByteDance. It generates high-quality audio-visual video captions. The repository includes code and checkpoints for multiple model sizes: video-SALMONN 2 (7B), video-SALMONN 2+ (3B, 7B, 72B), and video-SALMONN 2 Pro (4B, 8B, 32B). The models are evaluated on audio-visual QA benchmarks such as Video-MME, WorldSense, AVUT, Video-Holmes, and DailyOmni. The project is licensed under Apache-2.0.
Key features
- Generates high-quality audio-visual video captions.
- Available in multiple model sizes: 3B, 4B, 7B, 8B, 32B, and 72B.
- Supports audio input for enhanced understanding.
- Achieves state-of-the-art results on several audio-visual QA benchmarks at comparable scales.
- Includes training scripts for SFT (Supervised Fine-Tuning) and DPO (Direct Preference Optimization).
- Provides evaluation scripts for checkpoint assessment.
Use cases
- Generating detailed captions for videos that include both visual and audio content.
- Audio-visual question answering (QA) tasks.
- Research and development in multimodal large language models.
- Fine-tuning for specific audio-visual understanding applications.
Getting started
See the repository README for detailed instructions on how to use video-SALMONN 2, video-SALMONN 2+, and video-SALMONN 2 Pro. The README provides separate directories for each variant. For training, prepare datasets following the example JSON files (scripts/example_sft.json and scripts/example_dpo.json), download the LLaVA-OneVision model from Hugging Face, modify parameters in scripts/train_sft.sh or scripts/train_dpo.sh, and run the corresponding bash script. For evaluation, prepare the dataset, modify parameters in scripts/eval.sh, and run bash scripts/eval.sh.
License and pricing
The project is licensed under Apache-2.0. It is open source with no stated pricing.
Based on the project's public README and repository metadata.