caliper
Run your real agent with and without your skills, MCPs, and rules. See which ones actually help, and what they cost in tokens. Supports Claude Code, Codex, Pi,
What is caliper?
Run your real agent with and without your skills, MCPs, and rules. See which ones actually help, and what they cost in tokens. Supports Claude Code, Codex, Pi, Hermes, Antigravity, and OpenCode.
Overview of caliper
What it is
Caliper is a tool that evaluates agent skills by running the same tasks with and without the skill, measuring success rate and token cost. It supports Claude Code, Codex, Pi, and Hermes. The tool uses a spec file (.eval.yaml) that defines skills, tasks, and checks. Each task is run multiple times in a clean environment, and results are compared to determine if the skill actually helps.
Key features
- Real installs: Skills are installed where the agent looks, not pasted into the prompt.
- Activation scoreboard: Tracks whether the agent picked the skill, scored separately from task success.
- Neighbourhoods: Declare competing skills and assert which one should win each prompt.
- Ablation:
--ablatere-runs without the skill to see if it adds value. - Honest numbers: Uses single-run success rate first; pass@k can flatter flaky skills.
- Four agents: One spec works with Claude Code, Codex, Pi, and Hermes, chosen at runtime.
Use cases
- Verifying that an agent skill works reliably across multiple runs.
- Regression testing after model updates or when other skills compete for the same prompt.
- Comparing the performance of competing skills in a neighbourhood.
- Ablation testing to determine if a skill is earning its context cost.
Getting started
bashnpx skills@latest add edonadei/caliper# then, in your agent:/grill-skill ./inbox-triage/SKILL.md# or using pipx:pipx install caliper-evalSPEC=inbox-triage/inbox-triage.eval.yamlcaliper run $SPEC --ablate inbox-triagecaliper run $SPECcaliper compare .caliper/results/inbox-triage/RUN_ID.json inbox-triage
License and pricing
MIT license. Open source.
Based on the project's public README and repository metadata.