rag-experiment-accelerator

The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure Cogniti

What is rag-experiment-accelerator?

The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure Cognitive Search and RAG pattern.

Overview of rag-experiment-accelerator

What it is

The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure AI Search and the RAG pattern. It helps researchers, data scientists, and developers test search and OpenAI hyperparameters, compare search strategies, and generate detailed reports.

Key features

  • Experiment Setup: Define and configure experiments with a range of search engine parameters, search types, query sets, and evaluation metrics.
  • Integration: Seamlessly integrates with Azure AI Search, Azure Machine Learning, MLFlow, and Azure OpenAI.
  • Rich Search Index: Creates multiple search indexes based on hyperparameter configurations.
  • Multiple Document Loaders: Supports loading via Azure Document Intelligence and basic LangChain loaders.
  • Custom Document Intelligence Loader: Formats tables into key-value pairs, excludes irrelevant parts, and removes recurring patterns.
  • Query Generation: Generates diverse and customizable query sets for experimentation.
  • Multiple Search Types: Supports pure text, pure vector, cross-vector, multi-vector, hybrid, and more.
  • Sub-Querying: Breaks complex user queries into smaller sub-queries for relevant context.
  • Re-Ranking: Re-evaluates query responses using LLM and ranks by relevance.
  • Metrics and Evaluation: Supports end-to-end metrics (distance-based, cosine, semantic similarity) and component-based metrics (context recall, answer relevance) and retrieval metrics (MAP@k).
  • Report Generation: Automates report generation with visualizations.
  • Multi-Lingual: Supports language analyzers for individual languages and specialized analyzers.
  • Sampling: Creates a small but representative sample of data to speed up experimentation.

Use cases

  • Testing the performance of different Search and OpenAI related hyperparameters.
  • Comparing the effectiveness of various search strategies.
  • Fine-tuning and optimizing parameters.
  • Finding the best combination of hyperparameters.
  • Generating detailed reports and visualizations from experiment results.

Getting started

Clone the repository and set up the environment:

git clone https://github.com/microsoft/rag-experiment-accelerator.git
cd rag-experiment-accelerator
conda create -n rag-experiment python=3.11
conda init bash
### Close and reopen terminal, then:
conda activate rag-experiment
pip install .
az login
az account set --subscription=""
az account show

For development container setup, see the repository README.

License and pricing

The project is open source. The license is not explicitly stated in the metadata (NOASSERTION).

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/microsoftrag-experiment-accelerator.svg)](https://yunodb.com/tools/microsoftrag-experiment-accelerator?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/microsoftrag-experiment-accelerator?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/microsoftrag-experiment-accelerator.svg" alt="Featured on YunoDB" height="20"></a>