midscene
AI-powered, vision-driven UI automation for every platform.
What is midscene?
AI-powered, vision-driven UI automation for every platform.
Overview of midscene
What it is
Midscene.js is an AI-powered, vision-driven UI automation tool for E2E testing. It uses a visual GUI Agent to observe screens, act on elements, and verify results. It supports web, mobile, and desktop platforms.
Key features
- Write test cases in natural language using YAML or JavaScript.
- Locate and interact with UI elements by their appearance and position from screenshots, without maintaining selectors.
- Verify visible UI state such as colors, highlights, layout, and content on canvas or native apps.
- Cross-platform support: Web (Playwright, Puppeteer, desktop Chrome), Android (ADB), iOS (WebDriverAgent), HarmonyOS (HDC), Desktop (macOS, Windows, Linux, RDP), and custom interfaces.
- Supports multiple AI models: GPT, Qwen, Doubao Seed, DeepSeek, Gemini, Kimi, and open-source options.
- Midscene Test framework with YAML test cases, TypeScript Nodes for reusable business logic, lifecycle hooks, retries, concurrency, and step-level reports.
Use cases
- E2E testing for web, mobile, and desktop applications.
- Testing visual appearance and behavior that traditional selectors cannot capture.
- Automating complex workflows that require visual verification.
- Integrating AI-driven UI automation into existing test suites (Playwright, Vitest).
Getting started
See the repository README for installation and quick-start instructions.
License and pricing
Licensed under MIT. Open source.
Based on the project's public README and repository metadata.