Scrapling

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ a

What is Scrapling?

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

Overview of Scrapling

What it is

Scrapling is an adaptive web scraping framework designed for Python. It handles tasks from a single request to a full-scale crawl. Its parser can automatically relocate elements when a website's structure changes, and its fetchers can bypass anti-bot systems like Cloudflare Turnstile. The framework includes a spider system for concurrent, multi-session crawls with pause/resume, automatic proxy rotation, and adaptive crawl speed.

Key features

me

  • Spiders: A crawling framework with Scrapy-like API, concurrent crawling, multi-session support, pause/resume, streaming mode, blocked request detection, AutoThrottle, robots.txt compliance, development mode, ready-made spider templates (CrawlSpider, SitemapSpider, XMLFeedSpider, CSVFeedSpider, ShopifySpider), link extraction, and built-in export (JSON/JSONL/CSV/XML).
  • Fetchers: Multiple fetcher classes including Fetcher (HTTP requests), DynamicFetcher (browser automation via Playwright), and StealthyFetcher (anti-bot bypass). All support session management, proxy rotation, domain/ad blocking, DNS leak prevention, remote browsers via CDP, background API capture, and async support.
  • Parser: An adaptive parser that can automatically re-locate elements when a website's structure changes, using methods like css() with parameters such as auto_save and adaptive.

Use cases

mesScrapling can be used for web scraping projects of any scale. It is suitable for extracting data from static and dynamic websites, bypassing anti-bot protections, crawling entire domains with concurrent requests, and managing proxy rotation. The framework can be used for monitoring website changes, data aggregation, price monitoring, and content extraction from various platforms.

Getting started

To install Scrapling, use pip:bashpip install scraplingFor a quick example using the StealthyFetcher:pythonfrom scrapling.fetchers import StealthyFetcherStealthyFetcher.adaptive = Truep = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)products = p.css('.product', auto_save=True)products = p.css('.product', adaptive=True)For full crawling, see the spider example in the repository README.

License and pricing

Scrapling is open source under the BSD-3-Clause license.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/d4vinciscrapling.svg)](https://yunodb.com/tools/d4vinciscrapling?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/d4vinciscrapling?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/d4vinciscrapling.svg" alt="Featured on YunoDB" height="20"></a>