awesome-evals

A curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.

What is awesome-evals?

A curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.

Overview of awesome-evals

What it is

A curated, opinionated, non-BS library of resources for building and evaluating AI agents. It includes papers, blog posts, talks, courses, tools, and benchmarks. Maintained by BenchFlow. Every entry is annotated and verified: it states what it is and why it belongs, URLs are checked, quotes are verbatim, and dead or abandoned tools are pruned. The collection was assembled through a depth-4 recursive citation crawl (11.6k papers, ranked by in-degree), targeted practitioner-web discovery, 47 talks and podcasts transcribed and deep-noted, and per-section gap audits with adversarial verification. It contains 443+ curated links and 143 deep reading notes. Markers: 🆕 = released/updated 2025–2026, ⚠️ = caveat.

Key features

  • Annotated and verified entries with explanations of relevance.
  • Depth-4 recursive citation crawl covering 11.6k papers.
  • 47 talks and podcasts transcribed with verbatim quotes and timestamps.
  • 443+ curated links and 143 deep reading notes.
  • Markers for new content (🆕) and caveats (⚠️).
  • Includes a playbook (PATTERNS.md) with real, runnable code and worked examples for LLM-as-judge, pass@k/pass^k, error analysis, trajectory and world-state grading, CI gating, verifiable rewards, and more.
  • Per-section gap audits with adversarial verification.

Use cases

  • Building and evaluating AI agents for research or production.
  • Understanding evaluation infrastructure, benchmarks, and RL environments.
  • Learning about LLM-as-judge, verifiers, and agent-specific evaluation.
  • Safety and adversarial evaluation (prompt injection, jailbreaks, etc.).
  • Reference for practitioners, researchers, and product builders working with AI agents.

Getting started

See the repository README for contribution guidelines and how to navigate the curated list. No installation commands are provided.

License and pricing

The license is not stated in the repository metadata (NOASSERTION). The project is open source (pricing_type: open_source).

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/benchflow-aiawesome-evals.svg)](https://yunodb.com/tools/benchflow-aiawesome-evals?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/benchflow-aiawesome-evals?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/benchflow-aiawesome-evals.svg" alt="Featured on YunoDB" height="20"></a>