tokio-prompt-orchestrator
The proxy in front of your LLM should not be the slow part. Rust LLM API proxy: +0.5 ms, 55 MB, same prompt twice costs one call.
What is tokio-prompt-orchestrator?
The proxy in front of your LLM should not be the slow part. Rust LLM API proxy: +0.5 ms, 55 MB, same prompt twice costs one call.