VLMEvalKit

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

What is VLMEvalKit?

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

Overview of VLMEvalKit

What it is

VLMEvalKit is an open-source evaluation toolkit for large vision-language models (LVLMs). It supports over 220 large multi-modality models (LMMs) and more than 80 benchmarks. The toolkit enables one-command evaluation without the need for heavy data preparation across multiple repositories. It adopts generation-based evaluation for all LVLMs and provides results using both exact matching and LLM-based answer extraction.

Key features

  • Supports evaluation of 220+ LMMs and 80+ benchmarks, including commercial APIs and open-source models.
  • One-command evaluation workflow that simplifies benchmark testing.
  • Generation-based evaluation with exact matching and LLM-based answer extraction.
  • Multi-node distributed inference support using LMDeploy or VLLM for large-scale or thinking models.
  • Improved handling for models with thinking mode via a custom split_thinking function.
  • Support for long response outputs (exceeding 16k/32k tokens) with TSV format to prevent data truncation.
  • Regular updates adding new models and benchmarks.

Use cases

  • Evaluating and comparing the performance of large vision-language models across diverse benchmarks.
  • Research and development of LVLMs, including testing new model variants.
  • Benchmarking commercial and open-source models for leaderboard submissions.
  • Analyzing model capabilities in vision-centric tasks such as VQA, video understanding, and physics reasoning.

Getting started

See the repository README for installation and quick start commands.

License and pricing

Licensed under Apache-2.0. The project is open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/open-compassvlmevalkit.svg)](https://yunodb.com/tools/open-compassvlmevalkit?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/open-compassvlmevalkit?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/open-compassvlmevalkit.svg" alt="Featured on YunoDB" height="20"></a>