web-llm
High-performance In-browser LLM Inference Engine
What is web-llm?
High-performance In-browser LLM Inference Engine
Overview of web-llm
What it is
WebLLM is a high-performance in-browser LLM inference engine that runs language model inference directly within web browsers using WebGPU hardware acceleration. It requires no server support and is fully compatible with the OpenAI API, allowing the use of the same API on open source models locally. WebLLM is a companion project of MLC LLM.
Key features
- In-browser inference with WebGPU acceleration, enabling LLM operations without server-side processing.
- Full OpenAI API compatibility, including streaming, JSON-mode, logit-level control, seeding, and more.
- Structured JSON generation with custom JSON schema support, implemented in the WebAssembly portion of the model library.
- Extensive model support natively including Llama 3, Phi 3, Gemma, Mistral, Qwen, and others.
- Custom model integration in MLC format for deploying user-specific models.
- Streaming and real-time interactions for chat completions.
- Web Worker and Service Worker support for offloading computations and managing model lifecycle.
Use cases
Building AI assistants and chatbots that run entirely in the browser, enabling privacy-preserving AI applications. Structured JSON generation for applications requiring formatted output. Chrome extensions that leverage LLM capabilities. Real-time interactive applications such as virtual assistants.
Getting started
Install via package manager: sh npm install @mlc-ai/web-llm # or yarn add @mlc-ai/web-llm # or pnpm install @mlc-ai/web-llm Or import via CDN: javascript import * as webllm from 'https://esm.run/@mlc-ai/web-llm'; Create an MLCEngine instance and load a model: typescript import { CreateMLCEngine } from '@mlc-ai/web-llm'; const engine = await CreateMLCEngine('Llama-3.1-8B-Instruct-q4f32_1-MLC', { initProgressCallback: (progress) => console.log(progress), }); Note: Loading models requires downloading and may take significant time on first run.
License and pricing
Licensed under Apache 2.0. Open source.
Based on the project's public README and repository metadata.