OmAgent
[EMNLP-2024] Build multimodal language agents for fast prototype and production
What is OmAgent?
[EMNLP-2024] Build multimodal language agents for fast prototype and production
Overview of OmAgent
What it is
OmAgent is a Python library for building multimodal language agents. It is designed to keep the library simple without too much overhead compared to other agent frameworks. The library wraps complex engineering such as worker orchestration, task queue, and node optimization behind the scenes, leaving a super-easy-to-use interface to define an agent. It also provides useful abstractions for reusable agent components, enabling the construction of complex agents by aggregating basic components. OmAgent includes features required for multimodal agents, such as native support for VLM models, video processing, and mobile device connection, making it easy for developers and researchers to build agents that can reason over text, image, video, and audio inputs.
Key features
- A flexible agent architecture that provides a graph-based workflow orchestration engine and various memory types enabling contextual reasoning.
- Native multimodal interaction support including VLM models, real-time API, computer vision models, and mobile connection.
- A suite of state-of-the-art unimodal and multimodal agent algorithms that go beyond simple LLM reasoning, e.g., ReAct, CoT, SC-CoT.
- Supports local deployment of models using Ollama or LocalAI.
- Fully distributed architecture with support for custom scaling. Also supports Lite mode, eliminating the need for middleware deployment.
Use cases
OmAgent can be used to build video QA agents that answer questions about uploaded videos, as demonstrated in the provided example projects. It can also be used to build a mobile personal assistant similar to Google Astral. Additionally, it supports agentic operators, which are reusable agentic workflows such as CoT and ReAct, allowing comparison of various reasoning agent operators with the same LLM choice and test datasets.
Getting started
Requirements: Python >= 3.10. Install omagent_core using pip: bash pip install omagent-core Or install the latest version from source: bash pip install -e omagent-core For a quick start, generate the container.yaml file: bash cd examples/step1_simpleVQA python compile_container.py Configure your LLM settings in configs/llms/gpt.yml and set your OpenAI API key: bash export custom_openai_key='your_openai_api_key' export custom_openai_endpoint='your_openai_endpoint' Run the simple VQA demo with a webpage GUI: bash cd examples/step1_simpleVQA python run_webpage.py Open the webpage at http://127.0.0.1:7860.
License and pricing
The license is Apache-2.0. The pricing type is open source.
Based on the project's public README and repository metadata.