video-SALMONN-2

video-SALMONN 2 is a powerful audio-visual large language model (LLM) that generates high-quality audio-visual video captions, which is developed by the Departm

What is video-SALMONN-2?

video-SALMONN 2 is a powerful audio-visual large language model (LLM) that generates high-quality audio-visual video captions, which is developed by the Department of Electronic Engineering at Tsinghua University and ByteDance.

Overview of video-SALMONN-2

What it is

video-SALMONN 2 is an audio-visual large language model (LLM) developed by the Department of Electronic Engineering at Tsinghua University and ByteDance. It generates high-quality audio-visual video captions. The repository includes code and checkpoints for multiple model sizes: video-SALMONN 2 (7B), video-SALMONN 2+ (3B, 7B, 72B), and video-SALMONN 2 Pro (4B, 8B, 32B). The models are evaluated on audio-visual QA benchmarks such as Video-MME, WorldSense, AVUT, Video-Holmes, and DailyOmni. The project is licensed under Apache-2.0.

Key features

  • Generates high-quality audio-visual video captions.
  • Available in multiple model sizes: 3B, 4B, 7B, 8B, 32B, and 72B.
  • Supports audio input for enhanced understanding.
  • Achieves state-of-the-art results on several audio-visual QA benchmarks at comparable scales.
  • Includes training scripts for SFT (Supervised Fine-Tuning) and DPO (Direct Preference Optimization).
  • Provides evaluation scripts for checkpoint assessment.

Use cases

  • Generating detailed captions for videos that include both visual and audio content.
  • Audio-visual question answering (QA) tasks.
  • Research and development in multimodal large language models.
  • Fine-tuning for specific audio-visual understanding applications.

Getting started

See the repository README for detailed instructions on how to use video-SALMONN 2, video-SALMONN 2+, and video-SALMONN 2 Pro. The README provides separate directories for each variant. For training, prepare datasets following the example JSON files (scripts/example_sft.json and scripts/example_dpo.json), download the LLaVA-OneVision model from Hugging Face, modify parameters in scripts/train_sft.sh or scripts/train_dpo.sh, and run the corresponding bash script. For evaluation, prepare the dataset, modify parameters in scripts/eval.sh, and run bash scripts/eval.sh.

License and pricing

The project is licensed under Apache-2.0. It is open source with no stated pricing.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/bytedancevideo-salmonn-2.svg)](https://yunodb.com/tools/bytedancevideo-salmonn-2?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/bytedancevideo-salmonn-2?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/bytedancevideo-salmonn-2.svg" alt="Featured on YunoDB" height="20"></a>