evaluate
🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
What is evaluate?
🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
Overview of evaluate
What it is
🤗 Evaluate is a library for evaluating machine learning models and datasets. It provides standardized implementations of dozens of popular metrics, comparisons, and measurements. The library is designed to work with multiple frameworks including Numpy, Pandas, PyTorch, TensorFlow, and JAX. It is part of the Hugging Face ecosystem and is licensed under Apache-2.0.
Key features
- Includes implementations of dozens of popular metrics covering NLP, Computer Vision, and dataset-specific metrics.
- Supports comparisons (measuring differences between models) and measurements (tools to evaluate datasets).
- Provides type checking to ensure input formats are correct for each metric.
- Each metric comes with a card describing values, limitations, ranges, and usage examples.
- Community metrics live on the Hugging Face Hub, and users can easily add their own metrics.
- Allows creating new evaluation modules and pushing them to a dedicated Space on the Hub using the
evaluate-cli createcommand.
Use cases
- Evaluating machine learning models across different frameworks (Numpy, Pandas, PyTorch, TensorFlow, JAX).
- Comparing the performance of multiple models using standardized metrics.
- Assessing datasets with measurements to understand their properties.
- Adding custom evaluation metrics for specific projects or collaboration via the Hugging Face Hub.
Getting started
Install 🤗 Evaluate in a virtual environment using pip:bashpip install evaluateTo list available evaluation modules:pythonevaluate.list_evaluation_modules()To load and use a metric:pythonaccuracy = evaluate.load('accuracy')results = accuracy.compute(predictions=predictions, references=references)To create a new evaluation module, first install the template dependencies:bashpip install evaluate[template]Then run:bashevaluate-cli create 'Awesome Metric'
License and pricing
The library is open source under the Apache-2.0 license.
Based on the project's public README and repository metadata.