OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
What is OSWorld?
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Overview of OSWorld
What it is
OSWorld is a benchmark for multimodal agents that perform open-ended tasks in real computer environments. It was presented at NeurIPS 2024. The project provides a standardized evaluation framework where agents interact with actual desktop operating systems (Ubuntu, Windows) through screenshots and low-level actions (mouse clicks, keyboard inputs). The benchmark includes a set of tasks that require agents to understand visual interfaces and execute multi-step operations in realistic settings.
Key features
- Supports multiple virtual machine providers: VMware, VirtualBox, Docker (with KVM), Modal, Daytona, and AWS.
- Provides a standardized environment for evaluating multimodal agents on open-ended computer tasks.
- Includes a benchmark with tasks that require Google account access and proxy configuration for full functionality.
- Offers parallel evaluation support through AWS, reducing evaluation time to within one hour.
- Provides pre-downloaded files for initial state setup.
- The environment can be installed as a standalone package (
desktop-env) without benchmark tasks.
Use cases
- Evaluating multimodal AI agents on real desktop operating systems.
- Benchmarking the performance of vision-language models in interactive computer tasks.
- Research on autonomous agents that navigate graphical user interfaces.
- Training and testing agents for desktop automation.
Getting started
bash# Clone the OSWorld repositorygit clone https://github.com/xlang-ai/OSWorld# Change directory into the cloned repositorycd OSWorld# Optional: Create a Conda environment for OSWorld# conda create -n osworld python=3.10# conda activate osworld# Install required dependenciespip install -r requirements.txtAlternatively, install the environment without benchmark tasks:bashpip install desktop-env
License and pricing
The project is licensed under Apache-2.0. It is open-source and free to use.
Based on the project's public README and repository metadata.