DreamLayer-Eval
DreamLayer Eval: open-source benchmarking for image and video diffusion models. Automate prompts, seeds, and metrics for reproducible results. A research projec
What is DreamLayer-Eval?
DreamLayer Eval: open-source benchmarking for image and video diffusion models. Automate prompts, seeds, and metrics for reproducible results. A research project from DreamLayer AI (dreamlayer.io).
Overview of DreamLayer-Eval
What it is
DreamLayer Eval is an open-source benchmarking and evaluation platform for image generation models and video generation models. It automates prompts, seeds, metrics, configs, and reproducible run logging so researchers and developers can compare model quality faster and more consistently. It runs locally with a React frontend, Flask-based services, SQLite run storage, and ComfyUI integration for image workflows.
Key features
- Automated Benchmarking: One run sweeps N prompts by M seeds by K samplers; metrics compute live during generation.
- Reproducibility by Default: Every run persists to SQLite with full configuration and metrics; replayable by run_id.
- Image and Video Metrics Built In: Includes CLIPScore, FID, LAION aesthetic, color harmony, sharpness, YOLOv8 composition F1 for images; FVD, SSIM, PSNR, LPIPS, temporal flickering, subject/background consistency, motion smoothness for video.
- Multi-Modal Today: Image and video evaluation available out of the box; audio benchmarking on roadmap.
- Reference-Free and Reference-Based: Works without ground-truth for many metrics; reference video unlocks SSIM, PSNR, LPIPS; FID uses reference set.
- Cached, Incremental, Comparable: Metrics persist per run; batch backfill endpoints; comparison API for side-by-side runs.
- Researcher-Friendly Exports: Run locally on own GPU; export to CSV or ZIP report bundle.
Use cases
- AI researchers comparing diffusion models across prompts, seeds, and metrics.
- ML Engineers evaluating image and video generation quality.
- Labs and teams building internal benchmarking workflows for generative models.
- Open-source model creators testing checkpoints, LoRAs, and workflows.
- Developers integrating custom metrics and evaluation pipelines.
- Benchmarking image generation model outputs, video generation model outputs, prompt-to-image alignment, image quality, aesthetic quality, object-level prompt adherence, temporal video consistency, and reference-based similarity.
Getting started
bash # Linux/macOS ./install_linux_dependencies.sh # or ./install_mac_dependencies.sh ./start_dream_layer.sh # Windows (PowerShell) Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass . .install_windows_dependencies.ps1 start_dream_layer.bat
License and pricing
License: GPL-3.0. This is an open-source project and free to use.
Based on the project's public README and repository metadata.