unitxt
π¦ Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI bench
What is unitxt?
π¦ Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking
Overview of unitxt
What it is
Unitxt is a Python library for enterprise-grade evaluation of AI performance. It provides a catalog of tools and data for end-to-end AI benchmarking. It supports evaluation of text, tables, vision, speech, and code in a unified framework. It is model agnostic, working with HuggingFace, OpenAI, WatsonX, and custom models. The library is designed for reproducibility with shareable, modular components.
Key features
- Evaluate text, tables, vision, speech, and code in one unified framework.
- Enterprise-ready with battle-tested components and an extensive catalog of benchmarks.
- Model agnostic: works with HuggingFace, OpenAI, WatsonX, and custom models.
- Reproducible: shareable, modular components ensure consistent results.
- Command-line interface for simple and multi-task evaluations.
- Python API for custom evaluation pipelines.
Use cases
- Benchmarking AI models across multiple modalities (text, tables, vision, speech, code).
- Running standard benchmarks like MMLU-Pro, Text2SQL, and tool calling.
- Evaluating custom datasets with any model using the Python API.
- Interactive exploration of datasets and benchmarks via the graphical user interface.
Getting started
Install Unitxt: bash pip install unitxt Run a simple evaluation: bash unitxt-evaluate --tasks 'card=cards.mmlu_pro.engineering' --model cross_provider --model_args 'model_name=llama-3-1-8b-instruct' --limit 10 For the interactive dashboard: bash pip install unitxt[ui] unitxt-explore
License and pricing
Apache-2.0 license. Open source.
Based on the project's public README and repository metadata.