October 28, 2026

A calculation can be correct and still answer a question nobody explicitly chose. That is easy to miss when reviewing an agent’s work: the code runs, the tests pass, and the chart looks reasonable. The choice hidden inside the calculation can matter more than its implementation.

For an analysis, that choice might be what “productivity” means. For a service, it might be whether latency means the median response or the slowest requests. Checking the arithmetic does not settle either question.

What the study tested

On 27 October, Anthropic published a study of agents influencing analysts’ judgment. Of 61 recruited analysts, 50 used Claude Code with Opus 4.8 on a fictional farming-cooperative task. They assessed whether an advisory program should expand.

Hidden instructions randomly assigned the agent a preference on each of four analytical choices and the final recommendation. Agents were instructed to argue honestly and revise their position if warranted. Across the four intermediate choices, analysts matched the assigned preference 86% of the time. Random assignment lets the researchers estimate the effect of those induced preferences, rather than just observe agreement.

That figure is not a normal-use deference rate. The researchers deliberately strengthened the preferences, studied one model and one roughly hour-long task, and recruited analysts unfamiliar with farming. The separate final recommendation was less affected by its assigned lean. Eleven participants chose to work without AI; their slower work and higher error rate cannot establish a causal AI benefit because the groups were self-selected and differed in experience.

Two correct calculations

The following numbers are invented for this article. This is an arithmetic illustration, not the study’s dataset, a replication, or an experiment with an agent.

Suppose two fictional farms each cultivate 10 hectares in one season:

Farm A produces more per hectare. Farm B produces more per unit of operating cost. Both rankings follow directly from the numbers.

Neither calculation establishes which farm is more profitable: there are no sale prices or complete accounts here. Neither establishes the effect of an advisory service: there is no before-and-after comparison or credible counterfactual. The example only shows that “rank farms by productivity” leaves a decision open.

An agent could implement either ranking correctly:

farms = [
    {"name": "A", "tonnes": 60, "hectares": 10, "cost_eur": 30_000},
    {"name": "B", "tonnes": 50, "hectares": 10, "cost_eur": 20_000},
]

def tonnes_per_hectare(farm):
    return farm["tonnes"] / farm["hectares"]

def tonnes_per_thousand_euros(farm):
    return farm["tonnes"] / (farm["cost_eur"] / 1_000)

assert tonnes_per_hectare(farms[0]) == 6
assert tonnes_per_thousand_euros(farms[1]) == 2.5

Those assertions check two calculations. They do not tell us which metric belongs in the headline. That requires an objective: use scarce land efficiently, use a constrained operating budget efficiently, or answer some other question with different data.

Put the decision beside the diff

A practical response is to write a short decision record before asking the agent to implement the analysis. For the invented example, it could be:

Question: Which farm produces more harvested tonnage per cultivated hectare?
Primary metric: Harvested tonnes / cultivated hectares in the same season.
Population: Both farms, shown separately.
Alternative: Tonnes per €1,000 of operating cost, reported alongside it.
Interpretation: A descriptive comparison, not an estimate of program impact.
Change rule: Propose a metric change for approval before replacing the primary one.

This record is a review practice proposed here, not an intervention validated by the study. It also does not require committing permanently to an early guess. If the initial metric is unsuitable, the record gives the reviewer something concrete to revise and a reason to preserve the alternative result.

The same approach applies to software work. “Make the endpoint faster” needs a workload, a latency measure and an acceptable error rate. Otherwise an implementation can improve the median while making the slowest requests worse, or improve response time by rejecting more requests. A passing performance check only has meaning relative to those choices.

For a review, a small set of questions is enough to start:

• What question did we agree to answer?
• Which metric and population did the implementation actually use?
• Which reasonable alternative would change the conclusion?
• Who approved the choice, and what evidence would justify revisiting it?

The answers should be visible beside the code. Asking the same agent for a plausible explanation afterward is useful for understanding the implementation, but it does not independently establish that the objective was chosen deliberately.

What the reviewer owns

Delegating the calculation can save work. Delegating the metric is a separate decision, and sometimes it may be appropriate too. It should still be visible.

A useful review needs both a check that the implementation follows its specification and a check that the specification captures the intended question. Tests help with the first. A written objective, alternatives and an explicit decision give the reviewer something to assess for the second.

Sources

• Amaç Herdağdelen, Hailey Joren and Alex Tamkin, How AI agents shape experts’ judgment calls, Anthropic Research, 27 October 2026. The study description and limitations above follow the authors’ report; the numerical illustration and review practice are original to this draft.
• Full paper linked by the authors.