unitxt

πŸ¦„ Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI bench

What is unitxt?

πŸ¦„ Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking

Overview of unitxt

What it is

Unitxt is a Python library for enterprise-grade evaluation of AI performance. It provides a catalog of tools and data for end-to-end AI benchmarking. It supports evaluation of text, tables, vision, speech, and code in a unified framework. It is model agnostic, working with HuggingFace, OpenAI, WatsonX, and custom models. The library is designed for reproducibility with shareable, modular components.

Key features

  • Evaluate text, tables, vision, speech, and code in one unified framework.
  • Enterprise-ready with battle-tested components and an extensive catalog of benchmarks.
  • Model agnostic: works with HuggingFace, OpenAI, WatsonX, and custom models.
  • Reproducible: shareable, modular components ensure consistent results.
  • Command-line interface for simple and multi-task evaluations.
  • Python API for custom evaluation pipelines.

Use cases

  • Benchmarking AI models across multiple modalities (text, tables, vision, speech, code).
  • Running standard benchmarks like MMLU-Pro, Text2SQL, and tool calling.
  • Evaluating custom datasets with any model using the Python API.
  • Interactive exploration of datasets and benchmarks via the graphical user interface.

Getting started

Install Unitxt: bash pip install unitxt Run a simple evaluation: bash unitxt-evaluate --tasks 'card=cards.mmlu_pro.engineering' --model cross_provider --model_args 'model_name=llama-3-1-8b-instruct' --limit 10 For the interactive dashboard: bash pip install unitxt[ui] unitxt-explore

License and pricing

Apache-2.0 license. Open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/ibmunitxt.svg)](https://yunodb.com/tools/ibmunitxt?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/ibmunitxt?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/ibmunitxt.svg" alt="Featured on YunoDB" height="20"></a>