trulens

Evaluation and Tracking for LLM Experiments and AI Agents

What is trulens?

Evaluation and Tracking for LLM Experiments and AI Agents

Overview of trulens

What it is

TruLens is an open-source evaluation and tracking tool for LLM experiments and AI agents. It is OpenTelemetry-native, meaning tracing is portable to any OTLP backend. The tool instruments any app with a decorator, scores every step with LLM judges that explain themselves, and allows comparing versions to ship the one that earns it. Evaluations can run either as traces land or over a dataset after the fact. TruLens is adopted by AI teams at Walmart Global Tech, Cisco, J.P. Morgan Chase, Equinix, VMware by Broadcom, Hitachi Digital Services, Thomson Reuters, phData, HID Global and others.

Key features

  • OpenTelemetry-based tracing: Every function call, LLM generation, retrieval, and tool invocation is captured as a structured OTEL span, making traces exportable to Jaeger, Grafana Tempo, Datadog, New Relic, or any OTLP-compatible backend.
  • Agentic evaluations: Seven purpose-built evaluators for agentic systems — LogicalConsistency, ExecutionEfficiency, PlanAdherence, PlanQuality, ToolSelection, ToolCalling, and ToolQuality — each measuring a distinct aspect of agent behavior.
  • Batch and inline evaluation: Run evaluations alongside your app, on existing data, or in offline batch mode using the Run API.
  • MCP support: Instrument Model Context Protocol tool calls with the MCP span type to capture tool name, arguments, output, and latency.
  • Selector API: Target any span attribute for evaluation using the flexible Selector API.

Use cases

TruLens is used to find where an agent fails and where cost can be cut without losing quality. It is applicable for tracing every step of an LLM application to identify the cause of a bad answer, comparing versions of an app to see the tradeoff between scores, latency, and cost, and evaluating agentic systems for reasoning coherence, execution efficiency, plan adherence, plan quality, tool selection, tool calling, and tool quality.

Getting started

Install the trulens pip package from PyPI:bashpip install trulensInstall with a specific LLM provider for feedback evaluation:bashpip install trulens trulens-providers-openai # OpenAI / Azure OpenAIpip install trulens trulens-providers-orcarouter # OrcaRouter (OpenAI-compatible gateway)pip install trulens trulens-providers-litellm # LiteLLM (Anthropic, Cohere, Mistral, …)pip install trulens trulens-providers-google # Google Geminipip install trulens trulens-providers-bedrock # AWS Bedrockpip install trulens trulens-providers-cortex # Snowflake Cortexpip install trulens trulens-providers-huggingface # HuggingFacepip install trulens trulens-providers-langchain # LangChain modelsInstall with a specific app framework integration:bashpip install trulens trulens-apps-langchain # LangChain / LangGraphpip install trulens trulens-apps-llamaindex # LlamaIndex

License and pricing

TruLens is open source under the MIT license. Pricing is not stated; the repository indicates it is open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/trueratrulens.svg)](https://yunodb.com/tools/trueratrulens?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/trueratrulens?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/trueratrulens.svg" alt="Featured on YunoDB" height="20"></a>