prompttools
Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).
What is prompttools?
Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).
Overview of prompttools
What it is
PromptTools is an open-source, self-hostable set of tools for experimenting with, testing, and evaluating prompts, large language models (LLMs), and vector databases. It is created by Hegel AI. The tools allow developers to evaluate using code, notebooks, and a local playground. The library does not forward LLM calls to a server; all source code executes on the user's machine. It does not store API keys or LLM inputs/outputs on a server; data stays local. Usage tracking via Sentry is included but can be opted out by setting the environment variable SENTRY_OPT_OUT.
Key features
- Supports experiments with multiple LLM APIs: OpenAI (Completion, ChatCompletion, Fine-tuned models), LLaMA.Cpp (LLaMA 1, LLaMA 2), HuggingFace (Hub API, Inference Endpoints), Anthropic, Mistral AI, Google Gemini, Google PaLM (legacy), Google Vertex AI, Azure OpenAI Service, Replicate, and Ollama (in progress).
- Supports experiments with vector databases and data utilities: Chroma, Weaviate, Qdrant, LanceDB, Pinecone (supported), Milvus (exploratory), Epsilla (in progress).
- Supports frameworks: LangChain, MindsDB (supported), LlamaIndex (exploratory).
- Supports computer vision: Stable Diffusion and Replicate's hosted Stable Diffusion.
- Provides a playground interface via Streamlit that can be run locally.
- Allows exporting experiment results to CSV, JSON, LoRA JSON, or MongoDB.
Use cases
Developers can use PromptTools to test prompts and parameters across different LLMs (e.g., OpenAI, Anthropic, LLaMA) and evaluate the retrieval accuracy of vector databases. It is suitable for prompt engineering, model comparison, and retrieval evaluation in AI applications.
Getting started
To install PromptTools, use pip:pip install prompttoolsTo run a local example with Jupyter notebook:git clone https://github.com/hegelai/prompttools.gitcd prompttools && jupyter notebook examples/notebooks/OpenAIChatExperiment.ipynbTo launch the playground interface:pip install notebookpip install prompttoolsgit clone https://github.com/hegelai/prompttools.gitcd prompttools && streamlit run prompttools/playground/playground.py
License and pricing
The license is Apache-2.0. The project is open-source and self-hostable; pricing is not stated beyond being open-source.
Based on the project's public README and repository metadata.
