OSWorld

[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

What is OSWorld?

[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Overview of OSWorld

What it is

OSWorld is a benchmark for multimodal agents that perform open-ended tasks in real computer environments. It was presented at NeurIPS 2024. The project provides a standardized evaluation framework where agents interact with actual desktop operating systems (Ubuntu, Windows) through screenshots and low-level actions (mouse clicks, keyboard inputs). The benchmark includes a set of tasks that require agents to understand visual interfaces and execute multi-step operations in realistic settings.

Key features

  • Supports multiple virtual machine providers: VMware, VirtualBox, Docker (with KVM), Modal, Daytona, and AWS.
  • Provides a standardized environment for evaluating multimodal agents on open-ended computer tasks.
  • Includes a benchmark with tasks that require Google account access and proxy configuration for full functionality.
  • Offers parallel evaluation support through AWS, reducing evaluation time to within one hour.
  • Provides pre-downloaded files for initial state setup.
  • The environment can be installed as a standalone package (desktop-env) without benchmark tasks.

Use cases

  • Evaluating multimodal AI agents on real desktop operating systems.
  • Benchmarking the performance of vision-language models in interactive computer tasks.
  • Research on autonomous agents that navigate graphical user interfaces.
  • Training and testing agents for desktop automation.

Getting started

bash# Clone the OSWorld repositorygit clone https://github.com/xlang-ai/OSWorld# Change directory into the cloned repositorycd OSWorld# Optional: Create a Conda environment for OSWorld# conda create -n osworld python=3.10# conda activate osworld# Install required dependenciespip install -r requirements.txtAlternatively, install the environment without benchmark tasks:bashpip install desktop-env

License and pricing

The project is licensed under Apache-2.0. It is open-source and free to use.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/xlang-aiosworld.svg)](https://yunodb.com/tools/xlang-aiosworld?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/xlang-aiosworld?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/xlang-aiosworld.svg" alt="Featured on YunoDB" height="20"></a>