mlx-vlm
MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.
What is mlx-vlm?
MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.
Overview of mlx-vlm
What it is
MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on Mac computers with Apple Silicon, using the MLX framework. It supports text, images, video, and audio inputs. The project is built by Neywa Labs.
Key features
- Supports inference and fine-tuning of multimodal models (text, image, video, audio).
- Provides a command-line interface (CLI) for generation tasks.
- Includes a Gradio-based chat UI (requires extra dependency).
- Offers a FastAPI server with continuous batching, automatic prefix caching (APC), and KV cache quantization.
- Supports speculative decoding with DFlash, DFlash2, DSpark, Gemma 4 MTP, Gemma 4 EAGLE-3, and MiniMax M3 EAGLE-3 drafter families.
- Includes agent skills bundles for common workflows (CLI inference, server inference, conversion/quantization, adding new models, benchmarking, contributing, and reproducible GitHub issues).
- Provides model-specific documentation for many supported models (e.g., LFM3, DeepSeek-OCR, Phi-4, Gemma 4, MiniCPM, etc.).
Use cases
- Running vision-language model inference on Apple Silicon Macs.
- Fine-tuning VLMs for custom tasks.
- Building local multimodal AI applications (chat, OCR, audio understanding, image generation, 3D Gaussian splats).
- Deploying a local server for multimodal inference with continuous batching and caching.
- Experimenting with speculative decoding to speed up generation.
Getting started
Install the package using pip:shpip install -U mlx-vlmFor the Gradio chat UI, install with the ui extra:shpip install -U 'mlx-vlm[ui]'
License and pricing
The project is licensed under the MIT License. It is open source and free to use.
Based on the project's public README and repository metadata.