neural-compressor

Overview of neural-compressor

What it is

Intel® Neural Compressor is an open-source Python library that provides popular model compression techniques for mainstream deep learning frameworks including PyTorch, TensorFlow, and JAX. It supports static quantization, dynamic quantization, SmoothQuant, weight-only quantization, quantization-aware training, mixed precision, and more. The library enables advanced quantization of large language models (LLMs) and vision-language models (VLMs) such as LLaMA, Qwen, DeepSeek, Flux, and FramePack through integration with AutoRound. It is tested extensively on Intel hardware (Gaudi AI Accelerators, Core Ultra Processors, Xeon Scalable Processors, Xeon CPU Max Series, Data Center GPU Flex Series, and Data Center GPU Max Series) and with limited testing on AMD CPU, ARM CPU, and NVIDIA GPU.

Key features

• Supports multiple quantization techniques: static, dynamic, SmoothQuant, weight-only, quantization-aware training, and mixed precision.
• Provides advanced low-bit quantization for LLMs and VLMs, including INT8, FP8, MXFP8, INT4, MXFP4, and NVFP4 data types.
• Integrates with AutoRound for automated quantization of large models.
• Works with PyTorch, TensorFlow, and JAX frameworks.
• Optimized for Intel hardware (Gaudi, Core Ultra, Xeon, Data Center GPU) with additional support for AMD, ARM, and NVIDIA platforms.

Use cases

• Quantizing and compressing deep learning models for deployment on Intel hardware, especially for LLM and VLM inference.
• Reducing model size and memory footprint while maintaining accuracy for production environments.
• Enabling FP8 quantization on Intel Gaudi AI Accelerators for faster LLM inference.
• Applying weight-only quantization to load large language models efficiently on Intel Gaudi2.

Getting started

Install the necessary framework extension: pip install neural-compressor-pt # for PyTorch pip install neural-compressor-tf # for TensorFlow pip install neural-compressor-jax # for JAX Example: FP8 quantization on Intel Gaudi2 python from neural_compressor.torch.quantization import ( FP8Config, prepare, convert, ) import torch import torchvision.models as models model = models.resnet18() qconfig = FP8Config(fp8_config='E4M3') model = prepare(model, qconfig) # Custom calibration (dummy example) model(torch.randn(1, 3, 224, 224).to('hpu')) model = convert(model) output = model(torch.randn(1, 3, 224, 224).to('hpu')).to('cpu') print(output.shape) For weight-only LLM loading on Intel Gaudi2: python from neural_compressor.torch.quantization import load model_name = 'TheBloke/Llama-2-7B-GPTQ' model = load( model_name_or_path=model_name, format='huggingface', device='hpu', torch_dtype=torch.bfloat16, )

License and pricing

Licensed under Apache-2.0. The library is open source and free to use.

Based on the project's public README and repository metadata.