How to Test a Client Support Agent with Hugging Face

Outcome

By the end of this tutorial, you will have a working three-layer evaluation pipeline for a client support agent: a private golden ticket suite on Hugging Face Hub, a Python task runner that logs traces and scores policy compliance, and an optional connection to τ²-bench for multi-turn simulation testing. You will know which metrics matter for support agents and how to wire a basic CI gate so regressions fail before they reach production.

Prerequisites

• Python 3.10 or later
• A Hugging Face account and a token with read/write access to datasets (huggingface.co/settings/tokens)
• A support agent you can call from Python (REST API, LangChain agent, or a simple wrapper around your LLM plus tools)
• Basic familiarity with pip, JSON, and pytest or GitHub Actions
• Optional: Docker, if you plan to run τ²-bench locally later

Install the core dependencies:

pip install datasets huggingface_hub requests pytest

Step 1: Define what "good" means for your support agent

Before touching Hugging Face, write down what success looks like. A support agent is not a Q&A bot. You are testing a loop: talk to the user, call the right tools, mutate state correctly, and stay inside business rules.

I advise scoring four dimensions on every test case:

1. Resolution — Was the ticket actually solved?
2. Tool accuracy — Did the agent call the right tools in a sensible order?
3. Policy compliance — Refund windows, identity verification, escalation rules
4. Conversation quality — Clear, empathetic, grounded in your knowledge base

Create a simple rubric file eval/rubric.json:

{
  "weights": {
    "resolution": 0.4,
    "tool_accuracy": 0.3,
    "policy": 0.2,
    "ux": 0.1
  },
  "hard_fail_on_policy_violation": true
}

This rubric drives every scorer you build in the steps below. Without it, you will end up grading polite answers that did the wrong thing in your CRM.

Step 2: Build your golden ticket dataset

Public benchmarks do not know your refund policy or your CRM schema. Start with 20 to 50 anonymized tickets that reflect real edge cases: cancellations, billing disputes, escalation triggers, and ambiguous user messages.

Create eval/golden_tickets.jsonl. Each line is one test case:

{
  "ticket_id": "CS-001",
  "user_message": "I want to cancel order #84920 and get a refund.",
  "customer_context": {"tier": "gold", "verified": true, "order_id": "84920"},
  "expected_tools": ["lookup_order", "cancel_order"],
  "must_not_do": ["refund_without_verification"],
  "expected_final_state": {"order_status": "cancelled"}
}

Upload this dataset to Hugging Face Hub so your team can version it like code:

import json
from datasets import Dataset, DatasetDict

rows = []
with open("eval/golden_tickets.jsonl") as f:
    for line in f:
        rows.append(json.loads(line))

ds = DatasetDict({"test": Dataset.from_list(rows)})
ds.push_to_hub("your-org/support-agent-golden-v1", private=True)

Replace your-org with your Hugging Face namespace. Set the repo to private if it contains client-specific scenarios.

Expected result: a dataset page on Hugging Face Hub with a test split you can load anywhere with one line of code.

Step 3: Write a trace logger for every eval run

Every evaluation run should produce a structured trace. You cannot debug "the agent felt wrong" without logs.

Create eval/trace.py:

import json
from dataclasses import dataclass, field, asdict
from datetime import datetime, timezone

@dataclass
class AgentTrace:
    case_id: str
    turns: list = field(default_factory=list)
    tool_calls: list = field(default_factory=list)
    final_state: dict = field(default_factory=dict)
    scores: dict = field(default_factory=dict)
    started_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())

    def save(self, path: str):
        with open(path, "a") as f:
            f.write(json.dumps(asdict(self)) + "\n")

Your agent wrapper should append each user message, assistant reply, and tool invocation to this trace object. The trace is the contract between your agent and every scorer in the pipeline.

Step 4: Implement deterministic scorers

Text-only grading misses the most common support agent failures. Start with checks you can run without an LLM.

Create eval/scorers.py:

def score_tools(trace, expected_tools: list[str]) -> float:
    called = [t["name"] for t in trace.tool_calls]
    if not expected_tools:
        return 1.0
    hits = sum(1 for tool in expected_tools if tool in called)
    return hits / len(expected_tools)

def score_policy(trace, must_not_do: list[str]) -> float:
    for rule in must_not_do:
        if rule == "refund_without_verification" and not trace.final_state.get("verified"):
            if any(t["name"] == "issue_refund" for t in trace.tool_calls):
                return 0.0
    return 1.0

def score_resolution(trace, expected_final_state: dict) -> float:
    if not expected_final_state:
        return 1.0
    matches = sum(
        1 for k, v in expected_final_state.items()
        if trace.final_state.get(k) == v
    )
    return matches / len(expected_final_state)

These functions are intentionally simple. Adapt the policy rules to match your business logic. The pattern matters more than the exact conditions: deterministic first, LLM judge second.

Step 5: Build the task runner

The runner loads cases from Hugging Face, executes your agent against a sandbox, and writes traces plus scores.

Create eval/run_eval.py:

from datasets import load_dataset
from trace import AgentTrace
from scorers import score_tools, score_policy, score_resolution

def run_case(case, agent_fn):
    trace = AgentTrace(case_id=case["ticket_id"])
    result = agent_fn(
        user_message=case["user_message"],
        context=case.get("customer_context", {}),
        trace=trace,
    )
    trace.final_state = result.get("final_state", {})
    trace.scores = {
        "tool_accuracy": score_tools(trace, case.get("expected_tools", [])),
        "policy": score_policy(trace, case.get("must_not_do", [])),
        "resolution": score_resolution(trace, case.get("expected_final_state", {})),
    }
    return trace

def main(return_traces=False):
    cases = load_dataset("your-org/support-agent-golden-v1", split="test")
    traces = [run_case(case, your_agent) for case in cases]
    for t in traces:
        t.save("eval/results/traces.jsonl")
        print(t.case_id, t.scores)
    if return_traces:
        return traces

if __name__ == "__main__":
    main()

Replace your_agent with a function that calls your real support agent. Never point evals at production CRM APIs. Use a sandbox or mock server that returns the same shapes as production.

Run it:

python eval/run_eval.py

Expected result: a traces.jsonl file with one JSON object per case, each containing tool calls, final state, and numeric scores.

Step 6: Add an LLM judge for conversation quality

Deterministic scorers catch wrong actions. They do not catch vague, unhelpful, or hallucinated replies. For that layer, use a Hugging Face model as a judge.

Create eval/judge.py:

import os
import requests

HF_TOKEN = os.environ["HF_TOKEN"]
API_URL = "https://api-inference.huggingface.co/models/mistralai/Mistral-7B-Instruct-v0.3"

def judge_reply(conversation: str, answer: str, reference: str | None = None) -> float:
    prompt = f"""Rate the support reply from 0 to 1 on clarity and helpfulness.
Conversation: {conversation}
Reply: {answer}
Reference (if any): {reference or "N/A"}
Return only a number between 0 and 1."""

    response = requests.post(
        API_URL,
        headers={"Authorization": f"Bearer {HF_TOKEN}"},
        json={"inputs": prompt, "parameters": {"max_new_tokens": 8, "temperature": 0}},
        timeout=60,
    )
    response.raise_for_status()
    text = response.json()[0]["generated_text"]
    try:
        return float(text.strip().split()[-1])
    except ValueError:
        return 0.5

Call this judge only on cases where deterministic scorers pass, or on a sampled subset to control cost. Store the UX score in trace.scores["ux"].

Step 7: Run simulation tests with τ²-bench

Your golden set tests your business. Public simulation benchmarks test whether the agent can handle generic support workflows: multi-turn dialogue, tool use, and policy documents in domains like retail or telecom.

τ²-bench is built for exactly this. Install it following the project README, then run a baseline evaluation:

tau2 run \
  --domain retail \
  --agent-llm gpt-4.1 \
  --user-llm gpt-4.1 \
  --num-trials 3 \
  --num-tasks 20

To evaluate your agent instead of a generic LLM agent, implement a custom agent adapter as described in the τ²-bench Agent Developer Guide. Your adapter wraps the same your_agent function from Step 5.

Track these metrics over time:

• Task success rate
• Pass@k (success across multiple trials of the same task)
• Tool error rate
• Average turns per task

Run τ²-bench weekly, not on every PR. It is slower and more expensive than your golden set, but it tells you whether a prompt change broke general support competence.

Step 8: Wire a CI gate

Connect the golden set to your pull request workflow so policy regressions fail before merge.

Create eval/test_golden.py:

from run_eval import main as run_all

def test_no_policy_regressions():
    traces = run_all(return_traces=True)
    for trace in traces:
        assert trace.scores["policy"] == 1.0, f"Policy fail: {trace.case_id}"
        assert trace.scores["resolution"] >= 0.8, f"Resolution fail: {trace.case_id}"

Add a GitHub Actions job:

name: support-agent-eval
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install datasets huggingface_hub requests pytest
      - run: pytest eval/test_golden.py
        env:
          HF_TOKEN: ${{ secrets.HF_TOKEN }}
          AGENT_API_URL: ${{ secrets.AGENT_SANDBOX_URL }}

Start with a hard gate on policy compliance only. Loosen or tighten resolution thresholds once you have a stable baseline score.

Step 9: Publish eval results back to Hugging Face

After each nightly run, push traces and aggregate scores to a results dataset. This gives you history without building a dashboard on day one.

import json
from datasets import Dataset

rows = []
with open("eval/results/traces.jsonl") as f:
    for line in f:
        rows.append(json.loads(line))

Dataset.from_list(rows).push_to_hub(
    "your-org/support-agent-eval-results",
    split="nightly",
)

Compare runs over time: did resolution drop after a model swap? Did tool accuracy improve after a prompt change? The dataset becomes your eval audit trail.

Pitfalls and troubleshooting

Grading only the final message. A common failure mode: the agent says "Your refund is on the way" while calling issue_refund on the wrong order. Always score tool calls and final state, not just text.

Hitting production APIs during evals. If your runner mutates live customer data, stop and switch to a sandbox. Eval traffic should never touch production CRM, billing, or ticket systems.

Using public benchmarks as your only test suite. τ-retail and similar domains are useful baselines, but they do not know your escalation matrix or SLA rules. Keep your private golden set as the primary gate.

LLM judge as the sole scorer. Judges drift with temperature and model version. Use them for UX and grounding checks, not for hard policy gates.

Recap

You now have a three-layer testing flow for a client support agent:

1. Golden tickets on Hugging Face Hub for fast, client-specific regression testing
2. A Python runner with deterministic scorers for tools, policy, and resolution
3. Optional τ²-bench simulation for multi-turn, domain-generic support competence

Next step: add five edge cases you have seen fail in production (angry users, ambiguous order IDs, requests that should escalate) to your golden set, then re-run python eval/run_eval.py and inspect the traces before wiring CI.