Scrapling
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ a
What is Scrapling?
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Overview of Scrapling
What it is
Scrapling is an adaptive web scraping framework designed for Python. It handles tasks from a single request to a full-scale crawl. Its parser can automatically relocate elements when a website's structure changes, and its fetchers can bypass anti-bot systems like Cloudflare Turnstile. The framework includes a spider system for concurrent, multi-session crawls with pause/resume, automatic proxy rotation, and adaptive crawl speed.
Key features
me
- Spiders: A crawling framework with Scrapy-like API, concurrent crawling, multi-session support, pause/resume, streaming mode, blocked request detection, AutoThrottle, robots.txt compliance, development mode, ready-made spider templates (CrawlSpider, SitemapSpider, XMLFeedSpider, CSVFeedSpider, ShopifySpider), link extraction, and built-in export (JSON/JSONL/CSV/XML).
- Fetchers: Multiple fetcher classes including Fetcher (HTTP requests), DynamicFetcher (browser automation via Playwright), and StealthyFetcher (anti-bot bypass). All support session management, proxy rotation, domain/ad blocking, DNS leak prevention, remote browsers via CDP, background API capture, and async support.
- Parser: An adaptive parser that can automatically re-locate elements when a website's structure changes, using methods like
css()with parameters such asauto_saveandadaptive.
Use cases
mesScrapling can be used for web scraping projects of any scale. It is suitable for extracting data from static and dynamic websites, bypassing anti-bot protections, crawling entire domains with concurrent requests, and managing proxy rotation. The framework can be used for monitoring website changes, data aggregation, price monitoring, and content extraction from various platforms.
Getting started
To install Scrapling, use pip:bashpip install scraplingFor a quick example using the StealthyFetcher:pythonfrom scrapling.fetchers import StealthyFetcherStealthyFetcher.adaptive = Truep = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)products = p.css('.product', auto_save=True)products = p.css('.product', adaptive=True)For full crawling, see the spider example in the repository README.
License and pricing
Scrapling is open source under the BSD-3-Clause license.
Based on the project's public README and repository metadata.