Chinese-LLaMA-Alpaca-2
中文LLaMA-2 & Alpaca-2大模型二期项目 + 64K超长上下文模型 (Chinese LLaMA-2 & Alpaca-2 LLMs with 64K long context models)
What is Chinese-LLaMA-Alpaca-2?
中文LLaMA-2 & Alpaca-2大模型二期项目 + 64K超长上下文模型 (Chinese LLaMA-2 & Alpaca-2 LLMs with 64K long context models)
Overview of Chinese-LLaMA-Alpaca-2
What it is
This project is the second phase of the Chinese LLaMA & Alpaca large model series, based on Meta's Llama-2. It releases Chinese LLaMA-2 base models and Chinese Alpaca-2 instruction-tuned chat models. The models expand and optimize the Chinese vocabulary of the original Llama-2, use large-scale Chinese data for incremental pre-training, and support FlashAttention-2 training. Standard models support 4K context length, while long-context versions support 16K and 64K context. RLHF series models are fine-tuned with human preference alignment.
Key features
- Expanded and optimized Chinese vocabulary (size: 55,296) for better Chinese text encoding/decoding - FlashAttention-2 for faster and more memory-efficient attention - Long context extension using PI (Position Interpolation) for 16K and YaRN for 64K context - Simplified Chinese-English bilingual system prompt for Alpaca-2 models - RLHF (Reinforcement Learning from Human Feedback) for improved value alignment - Supports 🤗transformers, llama.cpp, text-generation-webui, LangChain, privateGPT, vLLM
Use cases
- Chinese LLaMA-2 base models: text continuation, given preceding text, generate following text - Chinese Alpaca-2 chat models: instruction understanding, question answering, writing, chatting, interaction - Long-context models: tasks requiring long document understanding, such as privateGPT - RLHF models: scenarios requiring correct value expression
Getting started
See the repository README for detailed instructions on model download, quantization, and deployment.
License and pricing
Licensed under Apache-2.0. The project is open source.
Based on the project's public README and repository metadata.