smart-llm-loader
smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. Spend less time on preprocessing headaches and
What is smart-llm-loader?
smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. Spend less time on preprocessing headaches and more time building what matters. From RAG systems to chatbots to document Q&A, SmartLLMLoader handles the heavy lifting so you can focus on creating exceptional AI applications.
Overview of smart-llm-loader
What it is
smart-llm-loader is a Python package that transforms documents into LLM-ready chunks. It handles document processing tasks such as converting documents to markdown, OCR for scanned files, and context-aware text chunking. The package integrates with LangChain and LlamaIndex and is designed for use in RAG systems, chatbots, and document Q&A applications.
Key features
- Support for multiple LLM providers - In-built OCR for scanned documents and images - Flexible document type support - Supports different chunking strategies: context-aware chunking and page-based chunking - Supports custom prompts and custom chunking
Use cases
The package is suited for building RAG systems, chatbots, and document Q&A applications. It aims to reduce preprocessing effort, allowing developers to focus on application logic. The README mentions that its chunking approach has been benchmarked against traditional methods, with superior performance when paired with Google's Gemini Flash model.
Getting started
Install system dependencies (Poppler for PDF processing) and the package: bash # Ubuntu/Debian sudo apt-get install poppler-utils # macOS brew install poppler # Windows: download Poppler, extract, add bin to PATH # Install package pip install smart-llm-loader # or with Poetry poetry add smart-llm-loader Quick start example: python from smart_llm_loader import SmartLLMLoader # Set API key and model (e.g., Gemini, OpenAI, Anthropic) loader = SmartLLMLoader(file_path='your_document.pdf', chunk_strategy='contextual', model='gemini/gemini-1.5-flash') documents = loader.load_and_split() See the repository README for full parameter details.
License and pricing
The package is open source under the MIT license. Pricing is not stated; it is listed as open source.
Based on the project's public README and repository metadata.