GPTCache

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.

What is GPTCache?

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.

Overview of GPTCache

What it is

GPTCache is a library for creating a semantic cache for LLM queries. It is designed to reduce API costs and improve response times by caching responses from large language models. The project is fully integrated with LangChain and llama_index. A Docker image is available, allowing use from any programming language.

Key features

  • Semantic caching for LLM queries, aiming to slash API costs by 10x and boost speed by 100x (as stated in the README).
  • Fully integrated with LangChain and llama_index.
  • Supports both exact match and similar search caching.
  • Provides a temperature parameter to control cache skipping.
  • Uses embedding functions, data managers, and similarity evaluation for flexible cache configuration.
  • Docker image available for language-agnostic use.

Use cases

  • Reducing LLM API costs for applications with high traffic.
  • Improving response times by serving cached answers for repeated or similar queries.
  • Building scalable applications that rely on ChatGPT or other LLMs.

Getting started

Install via pip:

pip install gptcache

For development installation:

git clone -b dev https://github.com/zilliztech/GPTCache.git
cd GPTCache
pip install -r requirements.txt
python setup.py install

Before running examples, set the OPENAI_API_KEY environment variable. See the repository README for detailed usage examples.

License and pricing

License: MIT. Pricing: open source (free).

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/zilliztechgptcache.svg)](https://yunodb.com/tools/zilliztechgptcache?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/zilliztechgptcache?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/zilliztechgptcache.svg" alt="Featured on YunoDB" height="20"></a>