CyberScraper-2077

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

What is CyberScraper-2077?

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

Overview of CyberScraper-2077

What it is

CyberScraper-2077 is a web scraping tool that uses AI models (OpenAI, Gemini, and Ollama) to extract data from websites. It is open-source under the MIT license and is designed to provide a modern, efficient scraping experience. The tool is built with Python and requires version 3.10 or higher. It can be run locally or via Docker. The project repository is hosted on GitHub at https://github.com/itsOwen/CyberScraper-2077. The tool supports both a graphical interface via Streamlit and command-line usage. It also integrates with LiteLLM for additional model support. The README mentions a sponsor, Swiftproxy, which provides residential proxies, but this is not a feature of the tool itself.

Key features

  • AI-powered extraction using OpenAI, Gemini, and Ollama models.
  • Sleek Streamlit interface for user interaction.
  • Multi-format export: JSON, CSV, HTML, SQL, or Excel.
  • Tor network support for scraping .onion sites with automatic routing.
  • Stealth mode parameters to avoid bot detection.
  • Ollama support for open-source LLMs.
  • Async operations for fast scraping.
  • Smart parsing to structure scraped content.
  • Caching using LRU cache and custom dictionary to reduce API calls.
  • Upload extracted CSV data to Google Sheets with one click.
  • Bypass captcha by appending -captcha to the URL (native only, not on Docker).
  • Current browser feature uses local browser instance to bypass bot detections (use only when necessary).
  • Navigate through webpages and scrape data from different pages (BETA).

Use cases

CyberScraper-2077 can be used for data extraction from websites where traditional scraping methods may be insufficient. It is suitable for extracting structured data from pages that require AI understanding, such as product listings, search results, or content spread across multiple pages. The tool can handle paginated content and supports custom URL patterns for multi-page scraping. It is also useful for scraping .onion sites via Tor, and for scenarios where stealth and avoidance of bot detection are important. The ability to export to multiple formats makes it applicable for data analysis, reporting, or integration with other systems. The Google Sheets upload feature allows direct sharing of scraped data.

Getting started

  1. Clone the repository: bash git clone https://github.com/itsOwen/CyberScraper-2077.git cd CyberScraper-2077 2. Create and activate a virtual environment: bash virtualenv venv source venv/bin/activate 3. Install required packages: bash pip install -r requirements.txt 4. Install Playwright: bash playwright install 5. Set API keys: bash export OPENAI_API_KEY='your-api-key-here' export GOOGLE_API_KEY='your-api-key-here' 6. Run the Streamlit app: bash streamlit run main.py 7. Open browser to http://localhost:8501. For Docker installation, see the repository README.

License and pricing

The license is MIT. The tool is open-source and no pricing information is provided in the README or metadata.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/itsowencyberscraper-2077.svg)](https://yunodb.com/tools/itsowencyberscraper-2077?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/itsowencyberscraper-2077?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/itsowencyberscraper-2077.svg" alt="Featured on YunoDB" height="20"></a>