tokio-prompt-orchestrator
Tokio pipeline for serving LLM requests: bounded five-stage flow with dedup, circuit breakers, retries, rate limits and a dead-letter queue in front of Anthropi
What is tokio-prompt-orchestrator?
Tokio pipeline for serving LLM requests: bounded five-stage flow with dedup, circuit breakers, retries, rate limits and a dead-letter queue in front of Anthropic, OpenAI, llama.cpp or vLLM. REST, SSE, MCP, TUI.
