self-hosted-ai-stack

Deploy a self-hosted AI stack with Docker Compose: local LLMs, chat, document parsing, embeddings, transcription, text-to-speech and MCP tools. Powered by Ollam

What is self-hosted-ai-stack?

Deploy a self-hosted AI stack with Docker Compose: local LLMs, chat, document parsing, embeddings, transcription, text-to-speech and MCP tools. Powered by Ollama, LiteLLM, AnythingLLM and more. Local-first with auto-configured authentication, lightweight stacks, optional HTTPS and NVIDIA GPU support. MIT licensed. Multi-arch: amd64, arm64.

Overview of self-hosted-ai-stack

What it is

Self-Hosted AI Stack is a Docker Compose deployment that runs local AI models, provides a chat interface, document parsing, embeddings, transcription, text-to-speech, and MCP tools. It is powered by Ollama, LiteLLM, AnythingLLM, and other services. The stack is local-first, with auto-configured authentication, lightweight stack options, optional HTTPS via Caddy, and NVIDIA GPU support. It is MIT licensed and supports linux/amd64 and linux/arm64 architectures.

Key features

  • Zero-config: all services auto-configure on first start.
  • Secure by default: AnythingLLM password protection is enabled, and bundled API services auto-generate keys.
  • HTTPS-ready: optional Caddy overlay provides automatic TLS and binds direct HTTP ports to localhost.
  • Local by default: runs locally, with optional external provider support via GatewayCrate.
  • AI agent ready: connect goose on your workstation to GatewayCrate with the setup guide.
  • Flexible configuration: customize models, ports, providers, and API keys with simple env files.
  • Lightweight stacks: for lower memory requirements (as low as ~4.5 GB).
  • GPU acceleration: via NVIDIA CUDA.
  • Multi-architecture support: linux/amd64, linux/arm64.

Use cases

  • Web-based ChatGPT-like chat interface (chat-ui stack).
  • Speech-to-text → LLM → text-to-speech pipeline (voice-pipeline stack).
  • Chat UI with voice input/output (voice-chat stack).
  • Semantic search + LLM Q&A (rag-pipeline stack).
  • Document parsing + RAG (rag-pipeline-full stack).

Getting started

Requirements: A Linux server with Docker installed, at least 8 GB RAM (with small models). For larger LLM models (8B+), 16 GB or more is recommended.Start the full stack:git clone https://github.com/hwdsl2/self-hosted-ai-stackcd self-hosted-ai-stackdocker compose up -dPull a model (required before making LLM requests):docker exec ollama ollama_manage --pull llama3.2:3bRun the health check:./stack-check.shGet the GatewayCrate master key:docker exec litellm litellm_manage --showkeyRetrieve the auto-generated AnythingLLM password:docker exec anythingllm cat /app/server/storage/.initial_admin_passwordAccess AnythingLLM at http://<your-server-ip>:3001 and GatewayCrate Admin UI at http://<your-server-ip>:4000/ui.

License and pricing

MIT license. Open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/hwdsl2self-hosted-ai-stack.svg)](https://yunodb.com/tools/hwdsl2self-hosted-ai-stack?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/hwdsl2self-hosted-ai-stack?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/hwdsl2self-hosted-ai-stack.svg" alt="Featured on YunoDB" height="20"></a>