ChainForge

An open-source visual programming environment for battle-testing prompts to LLMs.

What is ChainForge?

An open-source visual programming environment for battle-testing prompts to LLMs.

Overview of ChainForge

What it is

ChainForge is an open-source visual programming environment for battle-testing prompts to large language models (LLMs). It provides a data flow interface for analyzing and evaluating LLM responses, enabling users to compare prompts, models, and response quality beyond ad-hoc chatting with individual LLMs. The tool is built on ReactFlow and Flask.

Key features

  • Query multiple LLMs at once to test prompt ideas and variations quickly.
  • Compare response quality across prompt permutations, models, and model settings.
  • Setup evaluation metrics (scoring functions) and immediately visualize results across prompts, prompt parameters, models, and model settings.
  • Use AI to streamline the process: create synthetic tables and input examples with built-in genAI features, or prompt a model to generate starter code for evaluations.
  • Supports multiple providers including OpenRouter, OpenAI, Anthropic, Google Gemini, DeepSeek, HuggingFace, Together.ai, Ollama API, Microsoft Azure OpenAI Endpoints, and Amazon Bedrock.
  • Supports retrieval-augmented generation (RAG) with nodes for upload, chunking, retrieval, and reranking.

Use cases

ChainForge is designed for prompt engineering and evaluation workflows. Use cases include comparing prompt variations across different LLMs, conducting ground truth evaluations using tabular data, testing model behavior with combinatorial prompt inputs, and setting up automated evaluation metrics to assess response quality.

Getting started

To install ChainForge locally, ensure Python 3.10 or higher is installed, then run:bashpip install chainforgeAfter installation, start the server with:bashchainforge serveOpen localhost:8000 in a supported browser (Google Chrome, Firefox, Microsoft Edge, or Brave). For Docker, use:bashdocker compose up -dAccess ChainForge at http://localhost:8000.

License and pricing

ChainForge is open-source under the MIT license. The pricing type is open_source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/ianarawjochainforge.svg)](https://yunodb.com/tools/ianarawjochainforge?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/ianarawjochainforge?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/ianarawjochainforge.svg" alt="Featured on YunoDB" height="20"></a>