Evaluate your agent

• Test agent behavior: Run agents against predefined test cases
• Measure performance: Use custom evaluators to score responses
• Track experiments: All results are logged to Langfuse for analysis
• Automate CI/CD: Integrate evaluations into your deployment pipeline

​

Prerequisites

1. Langfuse credentials configured:
   .env

   LANGFUSE_PUBLIC_KEY=pk-xxx
   LANGFUSE_SECRET_KEY=sk-xxx
   LANGFUSE_HOST=https://cloud.langfuse.com  # Optional, defaults to cloud
   
2. BB AI SDK installed in your project

​

Quick start

1. Initialize the evals folder. Run the CLI command to scaffold the evals structure:
   bb-ai-sdk evals init
   
   This creates an evals/ folder with:

   File	Purpose
   __init__.py	Initializes observability for your framework
   agents.py	Registers your task functions
   evaluators.py	Defines custom evaluators
   evals_config.yaml	Experiment configuration
   datasets/	CSV dataset files

   Configure observability in evals/__init__.py (match your FastAPI app’s service_name / environment; pin bb-ai-sdk==0.1.9). The CLI scaffold may name this helper init_observability():
   import os
   from bb_ai_sdk.observability import configure_observability
   
   
   def init_observability() -> None:
       # No fastapi_app in the eval runner process — same service_name as app.py
       configure_observability(
           service_name="my-agent-api",
           environment=os.getenv("APP_ENV", "development"),
           framework="agno",  # or "langchain" / "langgraph"
       )
   
   
   init_observability()
   
   Install the matching [instrument-agno] (or langchain/langgraph) extra. LangChain/LangGraph can also use callback handlers. See Observability.
   For more information about configuring observability, see the Observability documentation.
2. Register your agent task. Edit evals/agents.py to register your agent as a task function:
   agents.py

   from evals import register_task
   from src.agents.my_agent import create_agent
   
   # Create your agent instance
   agent = create_agent()
   
   @register_task("my_agent")
   def my_agent_task(*, item, **kwargs):
       """
       Task function for Langfuse experiments.
   
       Args:
           item: Langfuse dataset item with .input attribute
           **kwargs: Additional keyword arguments
   
       Returns:
           String result from agent execution
       """
       result = agent.run(item.input)
       return result.content
   
   The task name in @register_task("my_agent") must match the name field in your config file.
3. Create a custom evaluator. Edit evals/evaluators.py to define how responses are scored:
   evaluators.py

   from langfuse.experiment import Evaluation
   from evals import register_evaluator
   
   @register_evaluator("accuracy_evaluator")
   def accuracy_evaluator(
       *,
       input: str,
       output: str | None,
       expected_output: str | None = None,
       metadata: dict | None = None,
       **kwargs,
   ) -> Evaluation:
       """Check if output matches expected output."""
       if output is None or expected_output is None:
           return Evaluation(name="accuracy", value=0.0, comment="Missing output or expected")
   
       is_match = output.strip().lower() == expected_output.strip().lower()
       return Evaluation(
           name="accuracy",
           value=1.0 if is_match else 0.0,
           comment="Match" if is_match else "No match",
       )
   
   The custom evaluator name in @register_evaluator("accuracy_evaluator") must match the name used in your config file.
4. Configure the experiment. Edit evals/evals_config.yaml to define your evaluation:
   evals_config.yaml

   agents:
     - name: "my_agent"              # Must match @register_task name
       skipEval: false               # Set to true to skip this task
       dataset:
         name: "my_dataset"          # CSV file at evals/datasets/my_dataset.csv
       evaluators:
         - "accuracy_evaluator"      # Must match @register_evaluator name
   
5. Create a dataset. Add a CSV file at evals/datasets/my_dataset.csv:
   my_dataset.csv

   input,expected_output
   "What is 2+2?","4"
   "Hello, how are you?","I'm doing well, thank you!"
   "What is the capital of France?","Paris"
   
   Run evaluations with bb-ai-sdk evals run and view results in your Langfuse dashboard!

​

Registering task functions

​

Task function signature

• Accept keyword arguments including item (with .input attribute)
• Return a string result
  @register_task("task_name")
  def task_function(*, item, **kwargs) -> str:
      # item.input contains the test case input
      result = your_agent.invoke(item.input)
      return str(result)
  

​

Creating custom evaluators

​

Evaluator function signature

• Accept keyword arguments: input, output, expected_output, metadata
• Return a Langfuse Evaluation object with name, value (score), and optional comment
  from langfuse.experiment import Evaluation
  from evals import register_evaluator
  
  @register_evaluator("evaluator_name")
  def my_evaluator(
      *,
      input: str,
      output: str | None,
      expected_output: str | None = None,
      metadata: dict | None = None,
      **kwargs,
  ) -> Evaluation:
      # Calculate score (0.0 to 1.0)
      score = calculate_score(output, expected_output)
      
      return Evaluation(
          name="evaluator_name",
          value=score,
          comment="Optional explanation"
      )
  

​

Configuration

​

Configuration structure

agents:
  - name: "agent_name"            # Required: matches @register_task name
    skipEval: false               # Optional: skip this agent (default: false)
    dataset:
      name: "dataset_name"        # Required: CSV filename (without .csv)
    evaluators:                   # Optional: list of evaluator names
      - "evaluator_1"
      - "evaluator_2"

​

Dataset format

​

CSV structure

​

Example datasets

input,expected_output
"What is 2+2?","4"
"What color is the sky?","blue"
"How many days in a week?","7"

input,expected_output,metadata
"Tell me about Langfuse","Langfuse is an open source LLM ops platform.","{\"lang\":\"en\"}"
"How do I reset my password?","Navigate to Settings > Security > Reset Password","{\"category\":\"account\",\"priority\":\"high\"}"
"What are your business hours?","We're open Monday-Friday, 9am-5pm","{\"category\":\"general\",\"priority\":\"low\"}"

input
"Write a haiku about programming"
"Explain quantum computing in simple terms"
"Generate a product description for wireless headphones"

​

Running evaluations

​

Using the CLI

# Run with default config (evals/evals_config.yaml)
bb-ai-sdk evals run

# Run with custom config path
bb-ai-sdk evals run --config path/to/config.yaml

​

Using Python

# Run with default config
python -m evals

# Run with custom config
python -m evals --config path/to/config.yaml

​

How it works

1. Discover tasks and evaluators. The framework automatically imports evals.agents and evals.evaluators modules to discover registered functions.
2. Manage datasets. For each agent, the framework checks if the dataset exists in Langfuse. If not, it uploads the CSV file automatically.
3. Run the experiment. The framework calls the task function for each dataset item and captures the results as Langfuse traces.
4. Evaluate the results. Each evaluator runs on the task output, and the framework logs scores to Langfuse.

​

Troubleshooting

# evals_config.yaml
agents:
  - name: "my_agent"  # Must match decorator

# agents.py
@register_task("my_agent")  # Must match config
def my_task(*, item, **kwargs):
    ...

evals/
└── datasets/
    └── my_dataset.csv  # Must match config dataset.name

export LANGFUSE_PUBLIC_KEY=pk-xxx
export LANGFUSE_SECRET_KEY=sk-xxx

LANGFUSE_PUBLIC_KEY=pk-xxx
LANGFUSE_SECRET_KEY=sk-xxx

@register_task("my_agent")
def my_task(*, item, **kwargs):
    result = agent.run(item.input)
    # Ensure we return a string
    return str(result.content) if result.content else "No response"

​

Next steps