VLMEvalKit
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
What is VLMEvalKit?
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
Overview of VLMEvalKit
What it is
VLMEvalKit is an open-source evaluation toolkit for large vision-language models (LVLMs). It supports over 220 large multi-modality models (LMMs) and more than 80 benchmarks. The toolkit enables one-command evaluation without the need for heavy data preparation across multiple repositories. It adopts generation-based evaluation for all LVLMs and provides results using both exact matching and LLM-based answer extraction.
Key features
- Supports evaluation of 220+ LMMs and 80+ benchmarks, including commercial APIs and open-source models.
- One-command evaluation workflow that simplifies benchmark testing.
- Generation-based evaluation with exact matching and LLM-based answer extraction.
- Multi-node distributed inference support using LMDeploy or VLLM for large-scale or thinking models.
- Improved handling for models with thinking mode via a custom
split_thinkingfunction. - Support for long response outputs (exceeding 16k/32k tokens) with TSV format to prevent data truncation.
- Regular updates adding new models and benchmarks.
Use cases
- Evaluating and comparing the performance of large vision-language models across diverse benchmarks.
- Research and development of LVLMs, including testing new model variants.
- Benchmarking commercial and open-source models for leaderboard submissions.
- Analyzing model capabilities in vision-centric tasks such as VQA, video understanding, and physics reasoning.
Getting started
See the repository README for installation and quick start commands.
License and pricing
Licensed under Apache-2.0. The project is open source.
Based on the project's public README and repository metadata.