evalscope

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

What is evalscope?

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

Overview of evalscope

What it is

EvalScope is a one-stop LLM evaluation framework built by the ModelScope Community. It supports model capability evaluation, inference performance stress testing, and result visualization. The framework can be started with a single command.

Key features

  • Comprehensive Evaluation Benchmarks: Built-in multiple industry-recognized evaluation benchmarks including MMLU, C-Eval, GSM8K, and more.
  • Multi-modal and Multi-domain Support: Supports evaluation of various model types including Large Language Models (LLM), Vision Language Models (VLM), Embedding, Reranker, AIGC, and more.
  • System One Evaluation: Evaluate text Choice decision models via --eval-type systemone_api, with task instructions, few-shot examples and structured probabilities/confidence.
  • Multi-backend Integration: Seamlessly integrates multiple evaluation backends including OpenCompass, VLMEvalKit, RAGEval to meet different evaluation needs.
  • Agent Evaluation Mode: Drives benchmarks (e.g. GSM8K, AIME, SWE-bench Agentic) inside a controlled multi-turn AgentLoop with pluggable strategies and tools, plus Docker or E2B remote sandboxes; full per-sample Agent Trace is recorded and visualizable.
  • Inference Performance Testing: Provides powerful model service stress testing tools, supporting multiple performance metrics such as TTFT, TPOT.
  • Interactive Reports: Provides a Web Dashboard for multi-dimensional model comparison, report overview and detailed inspection.
  • Arena Mode: Supports multi-model battles (Pairwise Battle), intuitively ranking and evaluating models.
  • Highly Extensible: Developers can easily add custom datasets, models and evaluation metrics.

Use cases

EvalScope is designed for evaluating and benchmarking large models (LLM, VLM, AIGC). It can be used for model capability evaluation, inference performance stress testing, and result visualization. The framework supports multi-modal and multi-domain evaluation, agent evaluation, and arena mode for model comparison.

Getting started

bashpip install evalscopeevalscope eval --model your-model-name --api-url $OPENAI_API_BASE_URL --api-key $OPENAI_API_KEY --eval-type openai_api --datasets gsm8k --limit 5

License and pricing

The license is Apache-2.0. The pricing type is open_source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/modelscopeevalscope.svg)](https://yunodb.com/tools/modelscopeevalscope?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/modelscopeevalscope?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/modelscopeevalscope.svg" alt="Featured on YunoDB" height="20"></a>