smart-llm-loader

smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. Spend less time on preprocessing headaches and

What is smart-llm-loader?

smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. Spend less time on preprocessing headaches and more time building what matters. From RAG systems to chatbots to document Q&A, SmartLLMLoader handles the heavy lifting so you can focus on creating exceptional AI applications.

Overview of smart-llm-loader

What it is

smart-llm-loader is a Python package that transforms documents into LLM-ready chunks. It handles document processing tasks such as converting documents to markdown, OCR for scanned files, and context-aware text chunking. The package integrates with LangChain and LlamaIndex and is designed for use in RAG systems, chatbots, and document Q&A applications.

Key features

  • Support for multiple LLM providers - In-built OCR for scanned documents and images - Flexible document type support - Supports different chunking strategies: context-aware chunking and page-based chunking - Supports custom prompts and custom chunking

Use cases

The package is suited for building RAG systems, chatbots, and document Q&A applications. It aims to reduce preprocessing effort, allowing developers to focus on application logic. The README mentions that its chunking approach has been benchmarked against traditional methods, with superior performance when paired with Google's Gemini Flash model.

Getting started

Install system dependencies (Poppler for PDF processing) and the package: bash # Ubuntu/Debian sudo apt-get install poppler-utils # macOS brew install poppler # Windows: download Poppler, extract, add bin to PATH # Install package pip install smart-llm-loader # or with Poetry poetry add smart-llm-loader Quick start example: python from smart_llm_loader import SmartLLMLoader # Set API key and model (e.g., Gemini, OpenAI, Anthropic) loader = SmartLLMLoader(file_path='your_document.pdf', chunk_strategy='contextual', model='gemini/gemini-1.5-flash') documents = loader.load_and_split() See the repository README for full parameter details.

License and pricing

The package is open source under the MIT license. Pricing is not stated; it is listed as open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/drminglersmart-llm-loader.svg)](https://yunodb.com/tools/drminglersmart-llm-loader?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/drminglersmart-llm-loader?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/drminglersmart-llm-loader.svg" alt="Featured on YunoDB" height="20"></a>