LlamaFactory

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

What is LlamaFactory?

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Overview of LlamaFactory

What it is

LlamaFactory (LLaMA-Factory) is an open-source project for unified, efficient fine-tuning of 100+ large language models and vision-language models; its metadata describes it as an ACL 2024 work. It offers a zero-code CLI and Web UI for fine-tuning, and the README states it is used by Amazon, NVIDIA, Aliyun, etc. Documentation is published on Read the Docs, and the project also maintains an official blog.

Key features

  • Model coverage: LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen3, Qwen3-VL, DeepSeek, Gemma, GLM, Phi and others.
  • Integrated methods: (continuous) pre-training, (multimodal) supervised fine-tuning, reward modeling, PPO, DPO, KTO, ORPO.
  • Scalable resources: 16-bit full-tuning, freeze-tuning, LoRA, and 2/3/4/5/6/8-bit QLoRA via AQLM/AWQ/GPTQ/LLM.int8/HQQ/EETQ.
  • Advanced algorithms: GaLore, BAdam, APOLLO, Adam-mini, Muon, OFT, DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ, PiSSA.
  • Practical tricks: FlashAttention-2, Unsloth, Liger Kernel, KTransformers, RoPE scaling, NEFTune, rsLoRA.
  • Wide tasks: multi-turn dialogue, tool using, image understanding, visual grounding, video recognition, audio understanding.
  • Monitoring and inference: LlamaBoard, TensorBoard, Wandb, MLflow, SwanLab; OpenAI-style API, Gradio UI and CLI with a vLLM worker or SGLang worker.

Use cases

The project targets fine-tuning of LLMs and VLMs either locally or in the cloud. The README lists free cloud options: Colab (free), PAI-DSW (free trial) and AMD GPU Cloud (free credits). Documented task types include multi-turn dialogue, tool use, image understanding, visual grounding, video recognition and audio understanding. The README also tracks Day-0/Day-1 support for new models such as Qwen3, Qwen2.5-VL, Gemma 3, GLM-4.1V, InternLM 3, MiniCPM-o-2.6, Llama 3, GLM-4, Mistral Small, PaliGemma2 and Llama 4. Deployment is documented with an OpenAI-style API and vLLM, and SGLang is supported as an inference backend via infer_backend: sglang.

Getting started

The README excerpt does not include installation commands; it refers readers to the usage section of the repository and to the online documentation. Cloud starting points listed in the README are Colab (free), PAI-DSW (free trial) and AMD GPU Cloud (free credits). For local installation and quick-start commands, see the repository README.

License and pricing

LlamaFactory is released under the Apache-2.0 license. The metadata classifies it as open source, and no pricing is stated.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/hiyougallamafactory.svg)](https://yunodb.com/tools/hiyougallamafactory?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/hiyougallamafactory?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/hiyougallamafactory.svg" alt="Featured on YunoDB" height="20"></a>