prompt-cache

Cut LLM costs by up to 80% and unlock sub-millisecond responses with intelligent semantic caching.A drop-in, provider-agnostic LLM proxy written in Go with sub-

What is prompt-cache?

Cut LLM costs by up to 80% and unlock sub-millisecond responses with intelligent semantic caching.A drop-in, provider-agnostic LLM proxy written in Go with sub-millisecond response

Overview of prompt-cache

What it is

PromptCache is a lightweight, self-hosted semantic cache for GenAI workloads. It sits between an application and an LLM provider, detects similar prompts, and can return a previously cached response instead of making another model call. It is written in Go and designed as a drop-in, provider-agnostic proxy.

Key features

  • Semantic caching with a two-threshold matching strategy: high similarity for direct cache hit, low similarity for cache miss, and a gray zone for optional intent verification using a smaller model.
  • Supports multiple embedding/verification providers: OpenAI, Mistral AI, and Anthropic/Claude (via Voyage AI for embeddings).
  • Streaming support for /v1/chat/completions with OpenAI-compatible SSE chunks on cache hits.
  • Runtime configuration of thresholds and gray-zone verifier via PATCH endpoint.
  • Bearer-token authentication for management endpoints (/metrics, /v1/stats, /v1/config, /v1/cache, etc.).
  • Persistent caching with configurable TTL (default 24 hours) using BadgerDB.
  • Cache warming and observability endpoints.

Use cases

  • RAG applications with recurring internal questions.
  • AI agents with repeated reasoning or tool-use patterns.
  • Support bots receiving similar customer questions.
  • Any production LLM workload with repeated or near-repeated requests.

Getting started

Docker: bash git clone https://github.com/messkan/prompt-cache.git cd prompt-cache export EMBEDDING_PROVIDER=openai export OPENAI_API_KEY=your_key_here # Strongly recommended for every non-local deployment export API_AUTH_TOKEN=your-secret-token docker-compose up -d Run from source: bash git clone https://github.com/messkan/prompt-cache.git cd prompt-cache export EMBEDDING_PROVIDER=openai export OPENAI_API_KEY=your-openai-api-key export API_AUTH_TOKEN=your-secret-token ./scripts/run.sh Or: bash make run # or go build -o prompt-cache cmd/api/main.go ./prompt-cache OpenAI-compatible client example: python from openai import OpenAI client = OpenAI( base_url='http://localhost:8080/v1', api_key='your-openai-api-key', ) client.chat.completions.create( model='gpt-4o-mini', messages=[{'role': 'user', 'content': 'Explain quantum physics'}], )

License and pricing

PromptCache is licensed under the MIT License. It is open source and free to use. The license is provided on an 'AS IS' basis without warranty of any kind.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/messkanprompt-cache.svg)](https://yunodb.com/tools/messkanprompt-cache?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/messkanprompt-cache?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/messkanprompt-cache.svg" alt="Featured on YunoDB" height="20"></a>