caliper

Run your real agent with and without your skills, MCPs, and rules. See which ones actually help, and what they cost in tokens. Supports Claude Code, Codex, Pi,

What is caliper?

Run your real agent with and without your skills, MCPs, and rules. See which ones actually help, and what they cost in tokens. Supports Claude Code, Codex, Pi, Hermes, Antigravity, and OpenCode.

Overview of caliper

What it is

Caliper is a tool that evaluates agent skills by running the same tasks with and without the skill, measuring success rate and token cost. It supports Claude Code, Codex, Pi, and Hermes. The tool uses a spec file (.eval.yaml) that defines skills, tasks, and checks. Each task is run multiple times in a clean environment, and results are compared to determine if the skill actually helps.

Key features

  • Real installs: Skills are installed where the agent looks, not pasted into the prompt.
  • Activation scoreboard: Tracks whether the agent picked the skill, scored separately from task success.
  • Neighbourhoods: Declare competing skills and assert which one should win each prompt.
  • Ablation: --ablate re-runs without the skill to see if it adds value.
  • Honest numbers: Uses single-run success rate first; pass@k can flatter flaky skills.
  • Four agents: One spec works with Claude Code, Codex, Pi, and Hermes, chosen at runtime.

Use cases

  • Verifying that an agent skill works reliably across multiple runs.
  • Regression testing after model updates or when other skills compete for the same prompt.
  • Comparing the performance of competing skills in a neighbourhood.
  • Ablation testing to determine if a skill is earning its context cost.

Getting started

bashnpx skills@latest add edonadei/caliper# then, in your agent:/grill-skill ./inbox-triage/SKILL.md# or using pipx:pipx install caliper-evalSPEC=inbox-triage/inbox-triage.eval.yamlcaliper run $SPEC --ablate inbox-triagecaliper run $SPECcaliper compare .caliper/results/inbox-triage/RUN_ID.json inbox-triage

License and pricing

MIT license. Open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/edonadeicaliper.svg)](https://yunodb.com/tools/edonadeicaliper?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/edonadeicaliper?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/edonadeicaliper.svg" alt="Featured on YunoDB" height="20"></a>