trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
What is trafilatura?
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Overview of trafilatura
What it is
Trafilatura is a Python package and command-line tool for gathering text and metadata from the web. It performs web crawling, downloading, scraping, and extraction of main texts, metadata, and comments from raw HTML. The tool is designed to be modular and does not require a database. It focuses on actual content while avoiding noise from headers, footers, and boilerplate. The extractor balances precision and recall. Trafilatura is used by organizations including HuggingFace, IBM, Microsoft Research, NVIDIA, the Allen Institute for AI, Stanford, and the Internet Archive.
Key features
- Advanced web crawling and text discovery: support for sitemaps (TXT, XML) and feeds (ATOM, JSON, RSS); smart crawling and URL management (filtering and deduplication)
- Parallel processing of live URLs and offline input (previously downloaded HTML files and parsed HTML trees)
- Robust and configurable extraction of main text (own rule-based extractor with jusText and readability-lxml as fallbacks), metadata (title, author, date, site name, categories and tags), formatting and structure (paragraphs, titles, lists, quotes, code, line breaks, in-line text formatting), and optional elements (comments, links, images, tables)
- Optional add-ons: language detection, speed optimizations
- Multiple output formats: TXT, Markdown, CSV, JSON, HTML, XML, and XML-TEI
Use cases
Trafilatura is used for web data acquisition in research and production environments. It is suitable for building text databases, extracting content from web pages for natural language processing, and converting raw HTML into structured data for analysis. The tool is employed in academic domains for data acquisition and by various organizations for content extraction tasks.
Getting started
See the repository README for installation and usage instructions.
License and pricing
This package is distributed under the Apache 2.0 license. Versions prior to v1.8.0 are under GPLv3+ license.
Based on the project's public README and repository metadata.