From Benchmark to Enterprise Practice: Systematic Understanding of LLM Agent Evaluation

Traditional LLM evaluation usually faces a relatively simple question: given a model some input, can it generate the correct answer? Agents change this question. An Agent may need to plan, search, call tools, modify the environment, handle errors, and only produce results after dozens of steps. A correct final answer does not mean the process is reliable; a failed task does not mean every preceding step was worthless.

This article takes the ACL 2026 Findings paper A Survey on Evaluation of LLM-based Agents as its main thread, systematically organizing the objects, Benchmarks, environments, metrics, and development frameworks of Agent Evaluation, and further answers several more practical questions: How to test Conversational Agents? How to score Trajectories? Is Human-in-the-loop equivalent to human intervention? How to evaluate LLMs and Harnesses separately? When building a domain-specific Research Agent, should you use Langfuse or implement Eval completely yourself?

Paper source: Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2026. A Survey on Evaluation of LLM-based Agents. Findings of ACL 2026, pages 26690–26714. DOI: 10.18653/v1/2026.findings-acl.1330.

This article will clearly distinguish two types of content: organization of paper classifications and conclusions, as well as engineering practice recommendations based on these conclusions.

The Fundamental Difference Between Agent Eval and Ordinary LLM Eval

Ordinary LLM Eval is often abstracted as:

Input → Model → Output → Score

Agent Eval faces an execution trajectory that modifies the environment:

Task
  ↓
Observation → Planning → Action / Tool Call
     ↑                            ↓
     └────── New Environment State
                         ↓
                   Final Outcome

Therefore, Agent Eval must at least answer three categories of questions:

1. Is the result correct: Was the task ultimately completed?
2. Is the process reliable: Are planning, tool selection, parameters, state updates, and error recovery reasonable?
3. Is the system deployable: Are costs, latency, stability, safety, and compliance acceptable?

This is why checking only the final text produces misjudgments. An Agent might claim "the refund is complete," but the database hasn't changed at all; or it may have genuinely completed the refund but bypassed authentication, creating a serious security issue.

The Agent Eval Landscape from the Paper

Paper Figure 1 divides the field into four main areas: Agent fundamental capabilities, domain-specific Agents (vertical Agents), general Agents, and frameworks for developing and evaluating Agents.

Fundamental Capability Evaluation

The paper focuses on organizing four capabilities.

Planning and Multi-Step Reasoning

Here we evaluate whether an Agent can:

• Break down complex goals into subtasks;
• Maintain long-term plans;
• Track environment state;
• Adjust plans based on execution results;
• Recover after failures.

Representative work includes PlanBench, FlowBench, and NaturalPlan. Current models typically handle short-term tactical planning, but still struggle with long sequences, constrained strategic planning.

Function Calling and Tool Use

Tool use is not just "whether a function call was generated," but multiple sub-problems:

Identify whether a tool is needed
→ Select the correct tool
→ Extract and populate parameters
→ Execute the tool
→ Understand tool results
→ Decide on next steps

Early work like BFCL and ToolBench focused more on single-step function selection and parameter structure; later work began incorporating multi-turn, nested calls, implicit parameters, state dependencies, and real MCP Servers, making evaluation closer to actual Agents.

Self-Reflection

Self-Reflection evaluation assesses whether an Agent can understand external feedback, correct beliefs, and change subsequent actions. The key is not having the model generate a "reflection," but whether the reflection actually improves decision-making.

This direction still lacks highly standardized methods. Simply comparing the final correctness rate before and after reflection obscures factors like prompting technique, feedback quality, and task difficulty.

Memory

Memory Eval focuses on whether an Agent can correctly store, retrieve, and use information across long conversations, multiple Sessions, and ongoing tasks. It needs to distinguish:

• episodic memory: what happened in the past;
• semantic memory: facts already acquired;
• procedural memory: how to perform an operation.

Being able to find a sentence in long context is not the same as having reliable Agent Memory. The real difficulty is dynamic updates, handling factual conflicts, switching across tasks, and avoiding treating outdated information as current state.

Application-Oriented Agent Benchmarks

Web Agent

The evolution of Web Agents reflects the changes in overall Agent Eval: from static pages and next-step action prediction to long-process operations in real or sandboxed websites.

• Mind2Web is closer to static evaluation based on cached trajectories;
• WebArena provides a dynamic website environment that can change state;
• WebVoyager and similar work further emphasize visual input and real webpage interaction;
• Security-oriented Benchmarks begin checking policy compliance and risky operations.

Static environments are easy to scale, but cannot represent the chain reactions of errors. In dynamic environments, one wrong click changes the pages seen afterwards, errors accumulate along trajectories, which is more realistic for actual Agents.

Software Engineering Agent

SWE-bench combines real GitHub Issues, complete code repositories, executable environments, and tests. Its core structure is:

Issue + Repository + Agent Patch + Executable Tests

SWE-bench Verified improves reproducibility through manual cleaning and containerization. Subsequent Benchmarks begin covering longer, more complex multi-file tasks, CLI operations, test generation, and management decisions.

Scientific Agent

Scientific Agent Eval has expanded from knowledge question-answering to complete research processes, including:

• Research conception;
• Hypothesis and experiment design;
• Scientific code generation;
• Paper reproduction;
• Peer Review;
• End-to-end scientific discovery.

ScienceAgentBench, CORE-Bench, PaperBench, and SciCode respectively cover different stages. Scientific tasks are difficult to express with a single success rate, so they typically require hierarchical Rubrics, breaking down complex tasks into separately scorable research subtasks.

Conversational Agent

Conversational Agent Benchmarks do not evaluate "whether the conversation sounds natural," but test whether an Agent can complete real business tasks through multi-turn dialogue while correctly calling APIs and following policies.

You can map it directly to SWE-bench:

Take order cancellation as an example; a test typically includes:

1. Initial orders and user data;
2. A user goal that may not be completely expressed;
3. Callable query, verification, cancellation, and refund tools;
4. Business policies;
5. LLM-driven User Simulator;
6. Expected final state and allowable operation constraints.

τ-Bench uses retail, airline, and other scenarios to test user simulation, API tools, and policy compliance. τ²-Bench further adds dual control: both Agent and simulated user can modify shared environments through tools. For example, in technical support, the user doesn't just say "I've restarted the router," but actually modifies device state through simulated tools. IntellAgent attempts to automatically generate tests from database Schema and policy documents; ALMITA generates different dialogue branches through intermediate task graphs, then applies manual filtering.

This type of evaluation should be at least broken down into four items:

• Dialogue: whether intent is understood and necessary information is requested;
• Tools: whether selection, parameters, and return value handling are correct;
• Task: whether the final environment state is correct;
• Compliance: whether authentication, permissions, or business policies are bypassed.

Generalist Agent

Generalist Agents need to complete tasks across applications while combining reasoning, browsing, file processing, code execution, and tool calling. GAIA leans toward comprehensive reasoning and tool selection; OSWorld operates a complete computer environment through GUI; AppWorld operates applications mainly through code and structured APIs; platforms like HAL attempt to unify multiple Benchmarks.

They face the key challenge: different Benchmarks have inconsistent tools, environments, and Harness interfaces, making it difficult to compare the same Agent under identical conditions.

Is Figure 1 a Complete Checklist?

Figure 1 is a good "research landscape," but not a complete Eval Checklist. It answers "what capabilities to evaluate, what applications to evaluate, what representative Benchmarks exist," but does not fully expand on "how to judge whether an Agent is good or not."

Paper Section 5 presents five questions closer to Benchmark design:

Additionally, an Eval oriented toward practical deployment should incorporate cost, latency, stability, error recovery, reproducibility, and Harness attribution. Therefore, a more complete perspective is:

Capability layer: What can the Agent do
Task layer: What business domain does the Agent work in
Execution layer: How does the Agent complete tasks and where does it fail
System layer: Is the Agent efficient, stable, secure, and deployable

Three Evaluation Granularities

The paper roughly divides quality evaluation supported by development frameworks into three levels: Final Response, Stepwise, and Trajectory.

Final Response Evaluation

Evaluates only the final output, for example:

• Is the answer correct;
• Is it faithful to the source;
• Does the tone meet requirements;
• Does the final database state match.

This approach is cheap and easy to automate, suitable for large-scale regression testing, but cannot pinpoint issues in intermediate steps.

Stepwise Evaluation

Step-by-step checking:

• Is routing correct;
• Is the correct tool selected;
• Are parameters valid;
• Is Retrieval relevant;
• Does the current action advance the goal.

It facilitates error localization, but isolating steps completely carries risks. A step may appear redundant in isolation but be reasonable exploration when combined with subsequent states; a step may be legal in itself but cause safety issues when placed in the wrong sequence.

Trajectory-Based Assessment

Trajectory Eval treats the complete "observation—decision—tool invocation—state change" sequence as a whole for evaluation.

Reference-based

Predefine one or more approved trajectories and compare them with the Agent's actual trajectory. Common matching approaches include:

• Exact: Actions, parameters, and order are all identical;
• Partial: Only compare critical steps;
• Subset: Actual trajectory must contain all necessary steps;
• Ordered: Steps can be non-consecutive but must maintain critical order;
• Unordered: Only requires complete action set;
• Graph-based: Check whether necessary nodes are visited, legal branches are taken, and state transitions occur.

Reference-based is reproducible but easily misjudges "different but correct" paths. For workflows requiring identity verification before data retrieval, reference graphs are usually more suitable than a single action sequence.

Reference-free

Does not provide standard paths but lets rules, human review, or LLM Judge determine if the actual trajectory is correct based on Rubric:

• Is it coherent;
• Does it continuously advance the goal;
• Is tool selection reasonable;
• Is it inefficient or looping;
• Can it recover from errors;
• Does it violate policies.

It can accept novel paths, but the Judge may deem "seemingly reasonable but actually incorrect" actions as correct. Therefore, the LLM Judge should use a fixed model and Prompt, require citations of specific Step IDs, and use human samples for calibration.

In practice, the most reliable approach is typically hybrid evaluation:

Programmatic hard checks
├─ Final state
├─ Parameter Schema
├─ Permissions and policies
└─ Required steps

Reference-based constraints
├─ Critical ordering
├─ Valid state transitions
└─ Prohibited paths

Reference-free Judge
├─ Policy rationality
├─ Efficiency
├─ Explanation quality
└─ Error recovery

Safety violations should not incur only small average penalties. Serious incidents like unauthorized refunds or data leaks are better handled as Hard Gates, directly failing the task.

Supporting Capabilities, Human-in-the-loop, and Gym Environment

These three terms are easily confused.

Supporting Capabilities

Supporting Capabilities are not the Agent's abilities but auxiliary functions provided by the Eval platform to manage the evaluation process, for example:

• Dataset and experiment management;
• Trace and production monitoring;
• Manual annotation;
• Synthetic Data Generation;
• A/B Comparison;
• Generating regression samples from production failures.

These capabilities address "how to organize exams, grade papers, compare versions, and manage scores."

Human-in-the-loop

In this paper, Human-in-the-loop mainly does not mean "humans help the Agent complete tasks during each execution" but rather:

• Expert annotation of results;
• Humans establish Gold Dataset;
• Review low-confidence samples from LLM Judge;
• Filter and anonymize failure cases from production logs;
• Calibrate automatic evaluators.

If humans provide hints or take over during Agent execution, more accurate terms are Human Intervention, Human Escalation, or Human-Agent Collaboration.

Gym-like Environment

Gym-like Environment is the interactive sandbox where the Agent actually takes tests. A typical interface can be abstracted as:

observation = env.reset(task)

while not done:
    action = agent.act(observation)
    observation, reward, done, info = env.step(action)

It needs several characteristics:

• Interactive: Agent actions change the environment;
• Resettable: Each run returns to the same initial state;
• Executable: Tools, code, or clicks produce real results;
• Verifiable: Final state can be checked;
• Loggable: Complete trajectories can be used for subsequent analysis;
• Relatively isolated: Avoid test operations affecting real accounts and production data.

BrowserGym provides web environments, SWE-Gym provides code repositories and terminal environments, MLGym provides machine learning experiment environments. These concepts can be distinguished in one sentence:

Benchmark Task: What questions to test
Gym Environment: Where to take the test
Metric / Evaluator: How to grade papers
Supporting Capabilities: How to manage exams and grades

How to Separately Evaluate LLM and Harness

The paper points out that many Agent Benchmarks simultaneously change the backbone LLM and Agent Harness, making it difficult to attribute performance improvements. The Harness here includes:

• System Prompt;
• Planner, Router, and Executor;
• Memory and Context Management;
• Retrieval;
• Tool Schema and execution strategy;
• Retry, Error Recovery, and termination logic;
• Guardrails.

Note: Harness cannot obtain a meaningful end-to-end total score without an LLM. The value of Harness lies in helping a specific LLM complete tasks. The correct approach is controlled experiments.

Fix the model, only change Harness

Must fix:

• Specific model version and inference configuration;
• Task Dataset;
• Tool permissions and environment initial state;
• Token, time, step, and Retry Budget;
• Randomness and number of repeated runs.

Then compare:

Same LLM + Minimal Harness
Same LLM + Planner
Same LLM + Planner + Memory
Same LLM + Planner + Memory + Retry

These Ablation Studies can answer how much improvement each module brings and whether the gain justifies the additional cost.

Further validate across multiple models

Fixing only one LLM is insufficient because Harness and model may have strong interactions. Strong models may not need complex Planners, weak models may fail to follow complex orchestration. A more reliable design is a Model × Harness matrix:

At this point, should report:

• Average Harness improvement across different models;
• Interaction between Model and Harness;
• Task Success and pass@k;
• Cost per Successful Task;
• Latency, step count, and tool call count;
• Safety violations and error recovery rate.

Harness components can still be tested independently

Mechanical modules of Harness can be tested without calling real LLMs:

• Test Router with fixed input;
• Test timeout, retry, and idempotency with Mock Tool;
• Test Context Manager with constructed trajectories;
• Test termination and loop detection with state machines;
• Test parameter validation and permission control with fixed tool calls.

Therefore, the complete solution is:

Component-level deterministic testing
       +
Fixed LLM Harness Ablation
       +
Cross-validation across multiple LLMs
       +
End-to-end testing in real sandbox

Vertical-Domain Research Agent: Use Langfuse or Implement Eval Yourself

This is not an either-or choice. A more practical boundary is:

Implement evaluation semantics and Grader yourself; leverage existing platforms for Trace, Dataset, Experiment, annotation, and Dashboard.

Platforms like Langfuse, LangSmith, and Phoenix excel at:

• Collecting and viewing Trace;
• Managing Dataset;
• Running different configurations on the same dataset;
• Storing scores from code, human, and LLM Judge;
• Tracking cost and latency;
• A/B comparison;
• Feeding production Traces back as regression samples.

However, they cannot define for you: what constitutes "high-quality research," which sources must be found, how Citations should support Claims, and which errors must be directly blocked from publication. These are core knowledge of vertical products.

Evaluation Dimensions for Research Agent

A usable starting point is decomposing evaluation into five layers.

Retrieval

• Key source recall rate;
• Primary source proportion;
• Source authority;
• Coverage of counterevidence;
• Compliance with time range;
• Presence of duplicate or broken links.

Citation

• Citation Validity: Does the source actually exist;
• Citation Entailment: Does the source support the corresponding Claim;
• Citation Placement: Is the citation placed after the correct statement;
• Citation Completeness: Are key facts all evidenced.

Synthesis

• Does it distinguish between facts, inferences, and opinions;
• Does it address source conflicts;
• Does it express uncertainty;
• Does it actually answer the user's question;
• Does it avoid simply pasting multiple articles together.

Trajectory

• Does the Query gradually converge;
• Does it prioritize finding primary sources;
• Does it actually open and read sources;
• Whether to change strategy after search failure;
• Whether to repeat the search;
• Whether to stop promptly after reaching sufficient evidence.

System

• Completion time;
• Token and API costs;
• Number of searches and tool calls;
• Stability across multiple runs;
• Security, privacy, and Prompt Injection risks.

Among these, issues like URLs, structure, numeric tolerance, and tool exceptions should be prioritized for deterministic code checks; then trade off comprehensive quality and evidence before handing over to domain experts or calibrated LLM judges.

A Minimal Viable Eval Project

You can start with the following structure:

eval/
├── datasets/
│   ├── golden_cases.jsonl
│   ├── adversarial_cases.jsonl
│   └── regression_cases.jsonl
├── graders/
│   ├── citation_validity.py
│   ├── citation_entailment.py
│   ├── source_quality.py
│   └── synthesis_judge.py
├── rubrics/
│   └── research_quality_v1.md
└── run_eval.py

Early prototypes can even use just JSONL, pytest, and a simple experiment report. Only integrate an evaluation platform when you need multi-person annotation, large-scale traces, continuous production monitoring, and version comparison. Don't mistake "having a pretty Trace UI" for "already having trustworthy Eval."

How Mature Enterprises Typically Do It

Mature teams usually adopt a hybrid architecture:

What enterprises truly accumulate long-term are usually three types of assets.

Golden Set

High-quality tasks confirmed by domain experts, including common tasks, high-value tasks, boundary conditions, and safety cases. The scale doesn't have to be huge, but each entry must have clear expected behavior and evaluation criteria.

Regression Set

Every production incident should be converted to a test that must pass in the future:

Online failure
→ Root cause analysis
→ Anonymize and minimize
→ Add to Regression Dataset
→ Fix Agent
→ Permanently prevent regression of similar issues

Eval Gate

Automatically run Eval when models, prompts, tool schemas, retrieval, memory, or planners change. Release conditions may include:

Core task success rate must not decrease
Key business Slice must not significantly degrade
Citation Correctness reaches threshold
Critical security violations are 0
P95 latency and average cost do not exceed budget

For real-world traffic post-launch with no ground truth, monitor through user feedback, tool errors, abnormal terminations, cost drift, citation spot-checks, LLM Judge sampling, and expert review. Online Eval detects new problems; offline Eval prevents problems from recurring.

Trends and Unresolved Issues Highlighted by the Paper

The paper summarizes two notable trends:

1. From simplified, static tasks to realistic, dynamic, long-horizon environments;
2. From easily outdated fixed benchmarks to continuously updated live benchmarks.

At the same time, several critical gaps remain:

• Coarse-grained success rate cannot pinpoint which step failed;
• Cost, latency, and resource consumption are not yet standard metrics;
• Manual benchmark construction is difficult to scale;
• Coverage of safety, robustness, and policy compliance is insufficient;
• Contributions of backbone LLM and harness are hard to decouple.

Together, these issues demonstrate: Agent Eval is not about choosing a benchmark and running out a score, but about establishing a continuously running quality engineering system.

An Actionable Agent Eval Checklist

If designing a domain-specific Agent Eval from scratch, answer these questions sequentially:

1. What task do users truly want to complete?
2. Can final success be verified through environment state or programmatic rules?
3. Which intermediate steps are necessary, and which paths are forbidden?
4. What interfaces does the agent use: code, tool APIs, or GUI?
5. How do you construct resettable, isolated dynamic environments?
6. Which metrics can be computed deterministically, and which need a judge?
7. Is the LLM Judge calibrated with human annotations?
8. Do you measure stability across multiple runs, not just one successful run?
9. Do you record tokens, latency, tool calls, and cost of successful tasks?
10. Is critical security violation set as a hard gate?
11. Do you fix the LLM and perform ablation on the harness?
12. Do you validate harness generalization across multiple models?
13. Can production failures automatically or semi-automatically flow back to the regression set?
14. Are dataset, rubric, evaluator, and harness all versioned?

Summary

The core of Agent Eval is not "does the final answer look right," but whether a system can reliably, economically, and compliantly complete objectives in dynamic environments.

The Figure 1 provided in the paper is suitable for building a domain map: planning, tools, reflection, memory, and web, SWE, research, dialogue, and general agents. But when actually designing evaluations, you must also add environment, interface, data, metrics, safety, trajectories, cost, and stability.

For domain-specific agents, the most realistic approach is not to hand all evaluation to Langfuse, nor to rewrite an entire platform from scratch, but to maintain your own dataset, rubric, and grader, while delegating common tracing, experimentation, annotation, and monitoring to mature infrastructure. In an enterprise environment, Eval ultimately becomes a closed loop: experts define quality, offline experiments prevent regressions, online monitoring detects new problems, and production failures are deposited as new regression tests.

In other words: a benchmark gives you a score; a mature Eval system tells you why it failed, what changed, whether you can ship, and where to focus next iteration.

References

• A Survey on Evaluation of LLM-based Agents
• Paper PDF
• Langfuse Evaluation Core Concepts
• LangSmith Evaluation Concepts
• Arize Phoenix Evaluation
• PaperBench: Evaluating AI's Ability to Replicate AI Research