opencompass

Overview of opencompass

What it is

OpenCompass is a one-stop platform for large model evaluation. It supports a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, and more, and integrates over 100 datasets covering knowledge, reasoning, coding, science, language, long context, safety, and other capability dimensions. OpenCompass also integrates VLMEvalKit, enabling text-only and vision-language datasets to be evaluated within a unified workflow.

Key features

• Supports a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, and more.
• Integrates over 100 datasets covering knowledge, reasoning, coding, science, language, long context, and safety.
• Integrates VLMEvalKit for unified evaluation of text-only and vision-language datasets.
• Provides a complete workflow from dataset and model configuration to inference, evaluation, and result summarization.
• Supports multiple inference backends including HuggingFace, LMDeploy, and vLLM.
• Supports zero-shot, few-shot, chain-of-thought, rule-based, and LLM-as-judge evaluation.
• Designed to be open, reproducible, and extensible.

Use cases

• Evaluating large language models across multiple capability dimensions.
• Comparing performance of different models on standardized benchmarks.
• Conducting research on model capabilities and safety.

Getting started

bash conda create --name opencompass python=3.12 -y conda activate opencompass pip install -U opencompass After installation, see the repository README for the Quick Start guide.

License and pricing

License: Apache-2.0. It is open source.

Based on the project's public README and repository metadata.