self-hosted-ai-stack
Deploy a self-hosted AI stack with Docker Compose: local LLMs, chat, document parsing, embeddings, transcription, text-to-speech and MCP tools. Powered by Ollam
What is self-hosted-ai-stack?
Deploy a self-hosted AI stack with Docker Compose: local LLMs, chat, document parsing, embeddings, transcription, text-to-speech and MCP tools. Powered by Ollama, LiteLLM, AnythingLLM and more. Local-first with auto-configured authentication, lightweight stacks, optional HTTPS and NVIDIA GPU support. MIT licensed. Multi-arch: amd64, arm64.
Overview of self-hosted-ai-stack
What it is
Self-Hosted AI Stack is a Docker Compose deployment that runs local AI models, provides a chat interface, document parsing, embeddings, transcription, text-to-speech, and MCP tools. It is powered by Ollama, LiteLLM, AnythingLLM, and other services. The stack is local-first, with auto-configured authentication, lightweight stack options, optional HTTPS via Caddy, and NVIDIA GPU support. It is MIT licensed and supports linux/amd64 and linux/arm64 architectures.
Key features
- Zero-config: all services auto-configure on first start.
- Secure by default: AnythingLLM password protection is enabled, and bundled API services auto-generate keys.
- HTTPS-ready: optional Caddy overlay provides automatic TLS and binds direct HTTP ports to localhost.
- Local by default: runs locally, with optional external provider support via GatewayCrate.
- AI agent ready: connect goose on your workstation to GatewayCrate with the setup guide.
- Flexible configuration: customize models, ports, providers, and API keys with simple env files.
- Lightweight stacks: for lower memory requirements (as low as ~4.5 GB).
- GPU acceleration: via NVIDIA CUDA.
- Multi-architecture support: linux/amd64, linux/arm64.
Use cases
- Web-based ChatGPT-like chat interface (chat-ui stack).
- Speech-to-text → LLM → text-to-speech pipeline (voice-pipeline stack).
- Chat UI with voice input/output (voice-chat stack).
- Semantic search + LLM Q&A (rag-pipeline stack).
- Document parsing + RAG (rag-pipeline-full stack).
Getting started
Requirements: A Linux server with Docker installed, at least 8 GB RAM (with small models). For larger LLM models (8B+), 16 GB or more is recommended.Start the full stack:git clone https://github.com/hwdsl2/self-hosted-ai-stackcd self-hosted-ai-stackdocker compose up -dPull a model (required before making LLM requests):docker exec ollama ollama_manage --pull llama3.2:3bRun the health check:./stack-check.shGet the GatewayCrate master key:docker exec litellm litellm_manage --showkeyRetrieve the auto-generated AnythingLLM password:docker exec anythingllm cat /app/server/storage/.initial_admin_passwordAccess AnythingLLM at http://<your-server-ip>:3001 and GatewayCrate Admin UI at http://<your-server-ip>:4000/ui.
License and pricing
MIT license. Open source.
Based on the project's public README and repository metadata.