TrustLLM
[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models,
What is TrustLLM?
[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.
Overview of TrustLLM
What it is
TrustLLM is an open research toolkit for evaluating the trustworthiness of large language models across six dimensions, as presented at ICML 2024. It supports local model weights, compatible APIs, and AI agent orchestration. The toolkit provides a Python/CLI interface to run the benchmark, keeping data, settings, and results together.
Key features
- Evaluates six trustworthiness dimensions: truthfulness, safety, fairness, robustness, privacy, and ethics.
- Supports both local model weights (via Hugging Face Transformers) and API-based models (OpenAI-compatible endpoints).
- Provides a unified CLI (
python -m trustllm) with commands:download,tasks,generate,evaluate. - Includes a Python API for programmatic use.
- Generates HTML reports for generation and scoring results.
- Offers optional dependencies for local inference (
[local]) and scoring ([eval]). - Bounded API retries, per-sample checkpoints, and resume support.
Use cases
- Evaluating the trustworthiness of a new or existing LLM across multiple dimensions.
- Reproducing the ICML 2024 benchmark on custom models or APIs.
- Integrating trustworthiness evaluation into an AI agent pipeline.
- Research on LLM safety, fairness, privacy, and ethics.
Getting started
bash python -m pip install 'trustllm @ git+https://github.com/HowieHwong/TrustLLM.git@main#subdirectory=trustllm_pkg' python -m trustllm download --output data python -m trustllm tasks export OPENAI_API_KEY='your-api-key' export OPENAI_BASE_URL='https://your-provider.example/v1' python -m trustllm generate --backend api --model your-model-id --task safety --data data/dataset --limit 5 --concurrency 4 --output runs/api-safety For local weights: bash python -m pip install 'trustllm[local] @ git+https://github.com/HowieHwong/TrustLLM.git@main#subdirectory=trustllm_pkg' python -m trustllm generate --backend local --model Qwen/Qwen2.5-0.5B-Instruct --task safety --data data/dataset --device auto --limit 5 --output runs/local-safety
License and pricing
The code is licensed under MIT. The dataset terms remain with the original sources. The toolkit is open source and free to use.
Based on the project's public README and repository metadata.
