Honey, I Looked at the Data of a Frontier Benchmark and Found some Issues: Porting Agents' Last Exam Linux CLI subset to Verifiers v1

16 Sep, 2026

Intro

2 months back, I started the Prime Residency with Florian Brand as my peer or shall I say my verifier. My first task was to port the Agents’ Last Exam Linux CLI subset to verifiers. Then one day Florian asked me to look into the data and this quest slowly evolved into finding issues in the benchmark and the making of what we call ALE-Gold.

It was more challenging than I expected. I was new to this kind of work, so there was some skill issue on my part. Along the way, I also hit benchmark defects, inference provider failures, and gaps in the freshly released Verifiers v1 (that have been fixed now so you don't need to worry). This post is a work log of my porting work, what the full runs revealed about the benchmark, how that led to ALE-Gold, followed by an analysis of how the models performed.

A small odyssey from porting ALE to ALE-Gold, with a few unexpected detours.

Table of contents

• Porting ALE Linux CLI subset to verifiers
  • What is a Rollout?
  • What it took to port ALE
    • One must minimise the rollout time
    • Missing dependency specifications
  • Full eval runs
• Look at the data, they said
  • Underspecified task contracts and evaluator defects
  • Answer Leakage and network issues
  • Broken inputs and missing dependencies
  • Missing or corrupt reference data
• ALE-Gold evaluation results
  • Sol (xhigh) vs Luna (xhigh)
  • Selected task comparisons
  • Failure modes
  • Performance by domain
  • Engineering has room for improvement
• Conclusion
• Acknowledgements

Porting ALE Linux CLI subset to verifiers

Agents’ Last Exam is considered a frontier benchmark made by Berkeley RDI, Dawn Song’s group at UC Berkeley. They worked with more than 300 experts across 100+ institutions to make long horizon economically valuable tasks spanning over 55 domains. Of the 1,500+ tasks in the corpus, 147 are public.

ALE was also featured in OpenAI’s GPT-6 Astra launch, where Astra scored 59.3% on Agents’ Last Exam.

My task was to port the Linux CLI subset, which consisted of 105 terminal-only tasks spanning 41 subdomains across 12 broad domain categories.

Why port to verifiers? ALE comes with its own evaluation implementation rather than being built on a standardized framework like Harbor or verifiers. Porting it to verifiers brings the tasks into a shared evaluation framework, with standardized rollout traces and a consistent way to configure models, harnesses, and runtimes. This lets others run their own evaluations using the same integration, and makes the resulting traces easier to inspect and compare.

One may wonder, “What does it mean to port a benchmark to verifiers?” Verifiers is a library by Prime Intellect that enables you to run “rollouts” in an easy and elegant manner. I mention the word rollout multiple times in this post so it's worth doing a small detour to understand it.

What is a Rollout?

A rollout in the context of evaluation and reinforcement learning is defined as a single run of a policy/model in an environment from start to finish, recording the states, actions, and rewards along the way.

For example when doing LLM-RL, you will be running multiple rollouts of a single task and these can have different scores/rewards. The policy parameters are updated using the spread of these rewards. If you use GRPO, differences in rewards within a group of rollouts provide a learning signal by showing which rollouts performed better relative to the others.

Same task, different attempts. Each rollout has its own sandbox and harness, takes a different path, and receives a score. Timelines and scores are illustrative.

If you look at benchmarks like FrontierSWE or TerminalBench (Harbor framework), they are usually divided into three parts - the task’s input in the form of files or a Docker image, a harness where the model will run like Claude Code and Codex and the runtime (where the rollout happens). The runtime can be your local machine, a Docker instance, or an AWS instance.

Verifiers provides this exact abstraction for you to port the benchmark and then run the benchmark on Prime Intellect’s infra including support for several sandbox providers.

A taskset defines the work and scoring, a harness controls the agent’s interaction loop, and a runtime provides the execution environment.

What it took to port ALE

To give some context, here’s what the porting process generally looks like.

The porting work mainly involves porting the taskset/dataset to verifiers and then writing relevant adapter code between verifiers and the original public benchmark code (which I refer to as “upstream”) to support evaluation (as it’s defined by upstream). In short, port the input and hook up the scorer code. In addition to this, one may need to write some more compatibility related code depending on the benchmark. The “porting work” itself can vary depending on if the benchmark providers have provided sandbox images or not.

In ALE’s case, images were not provided so I had to make sandbox images for 105 tasks with the mentioned dependency specifications and input files. This was a code intensive and time consuming process and was only made possible due to coding agents.

One must minimise the rollout time

Once the images were made, I would run a couple of rollouts of a task to verify if the image was fine and the rollout happened successfully.

In my first attempt at building the images, I wasn’t baking the “input files” into them. Instead, I was downloading them at runtime, which meant fetching them from the Hugging Face archive during every rollout. Now, most of the input files were small, but for a few tasks, they were several GBs in size. Halfway through, I started hitting HF’s resolver limits…

This is when I realised my mistake and discussed it with Florian too. I was supposed to bake the inputs onto the image. Baking the inputs into the image adds storage cost, but saves time and network requests because the files no longer need to be downloaded during each rollout.

From an evaluation point of view, it’s a redundant operation and in general people make an effort to minimise the rollout time.

From an RL post-training point of view, this is an even bigger issue because you run multiple rollouts of the same task repeatedly across several training steps, so downloading the same input files for every rollout adds up quickly. RL researchers and engineers strive to reduce rollout time, often by increasing inference throughput, because rollout generation is frequently the biggest bottleneck. In a synchronous RL training setup, you need to wait for all rollouts to finish before running the gradient steps so the slowest rollout becomes the bottleneck. You can read a bit about it here (on how Prime Intellect engineers improve inference throughput).

Missing dependency specifications

Another challenge was that the task cards did not always list everything needed to run the supplied software. In physical_sciences/glm_lake_calibration, the card listed Python and pointed to a bundled lake-simulation binary, but declared no required system packages. The binary still depended on four libraries that were missing from the image: libgd3, libgfortran5, libgomp1, and libx11-6. My agent added those dependencies to the image and included a check for unresolved libraries.

Other failures came from images not matching what the task promised. In physical_sciences/exact_diag_heisenberg_j1j2, the task specified a Python runtime with NumPy and SciPy already installed, but the image was missing that setup. The agent ended up downloading and unpacking the libraries during its rollout. After I added the expected runtime, packages, and wrapper to the image, a rerun scored 1.0 without any installation commands.

As I mentioned earlier, we want to minimise rollout time, so we should preinstall known dependencies rather than have the agent install them during each rollout.

I also adjusted the memory or disk requirements for eight tasks and configured their sandbox resources accordingly.

Full eval runs

Once the images were ready, I started doing full evaluation runs with different models and harnesses to verify my implementation. This is when you start realising how important reliable infrastructure is and how fast the money burns when you use models via API (Not that I didn't have this realisation earlier but we all have been running the hedonic treadmill thanks to Tibo's resets).

A full eval run with a model like GPT-5.6 Sol could take 6-12 hours, including retries. Some individual tasks ran for hours, and sandbox or inference provider failures meant retrying affected rollouts, making the feedback loop quite slow.

Occasionally, I ran into more specific issues. During my Kimi runs, failures in the ACP integration caused many rollouts to end midway through a task. This was total suffering.

Running these full evaluations also surfaced infrastructure issues, which I reported to the Prime team. Verifiers v1 had just been released, and these reports helped the team identify and address issues in the new release. I also dabbled a bit in the sandbox infra code to see if I could use a “separate sandbox judging runtime” for my benchmark. Some ALE tasks produced output artifacts larger than 1 GB, but transferring artifacts from the rollout sandbox to the judge sandbox was limited to 32 MB at the time. I found out the bottleneck was the sandbox SDK’s lack of support for chunked file streaming.

These runs also helped uncover issues in the benchmark itself. Florian also encouraged me to "look at the data". A closer look revealed more issues. Note: look at the data at your own risk. You never know what crimes you will uncover.

The next section is all about my suffering and the issues we found in the ALE Linux CLI subset and the fixes. 9 tasks had to be excluded because a fix wasn't possible from our side for them. The resulting subset, ALE-Gold, contains 96 of the original 105 tasks, spanning 40 subdomains.

Look at the data, they said

Before we’re too hard on the creators, I should mention that frontier benchmarks are hard to build. These tasks span specialized domains, involve complicated dependencies, and need verifiers that can distinguish a wrong answer from a valid solution the author didn’t anticipate. Getting all of this right takes a lot of care, both when building the tasks and when validating them.

With that context, here are some of the issues I found while inspecting the tasks and rollout traces.

Underspecified task contracts and evaluator defects

I observed agents were failing 5-6 tasks despite producing correct solutions because the task prompt / input didn’t specify the expected output format correctly. To be more specific, essential schemas, enums, or semantics were absent or ambiguous in the visible prompt. Below are a few examples.

>"frontier benchmark"
>rollout has partial score
>verifier is deterministic scorer
>look inside
>verifier expects output format with field names agent can't infer from task prompt even by hallucinating pic.twitter.com/qGpDddm1Mn

sankalp (@dejavucoder) August 18, 2026

if policy.get("empty_grade_behavior") == expected_policy["empty_grade_behavior"]:
    policy_points += 5

# NSCLC Group-Sequential Statistical Analysis Plan
You are given protocol.json for a two-arm Phase III advanced NSCLC trial.
Create a complete statistical-analysis-plan deliverable set under output/.
Required files:

SAP.md
analysis.R
sample_size.json
boundaries.csv
multiple_testing.json
power_curve.csv
power_curve.png
boundary_plot.png

Computational requirements:

Compute log-rank/Schoenfeld sample size for overall survival using the protocol hazard ratio, one-sided alpha, power, and target events.
Produce three-look O'Brien-Fleming efficacy boundaries at 50%, 75%, and 100% information.
Include binding futility only at look 1 using conditional power < 0.20; looks 2 and 3 should record futility as NA.
Report the sample-size inflation factor and adjusted total sample size.
Apply the gated Hochberg procedure to PFS, ORR, and DOR using familywise alpha 0.025, and report subgroup Bonferroni alpha.
Produce a power curve CSV and PNG over plausible hazard ratios including the target HR.
Produce a boundary plot PNG.
Write a full SAP narrative explaining the design assumptions, interim looks, multiplicity, and deliverables.

Rules:

Work from protocol.json; do not use internet access.
analysis.R should be a reproducible script for generating the outputs.
Numeric outputs should be rounded clearly but retain enough precision for verification.

    try:
      sample_size = _load_json(output_files["sample_size.json"])
      pred_n = _to_float(sample_size.get("per_arm_n"))
      ref_n = _to_float(ref["sample_size"]["per_arm_n"])
      if abs(pred_n - ref_n) / ref_n <= 0.03:
          checks["sample_size"] += 0.65
      for key in ["events_required", "total_n", "adjusted_total_n"]:
          if int(round(_to_float(sample_size.get(key)))) == int(ref["sample_size"][key]):
              checks["sample_size"] += 0.08
      inflation = _to_float(sample_size.get("inflation_factor"))
      if abs(inflation - _to_float(ref["sample_size"]["inflation_factor"])) <= 0.01:
          checks["sample_size"] += 0.11
      checks["sample_size"] = min(1.0, checks["sample_size"])
  except Exception as exc:
      issues.append(f"sample_size.json invalid: {exc}")

if isinstance(section.get("per_variable"), dict):
    entries.extend(section["per_variable"].items())
if isinstance(section.get("table"), list):
    entries.extend(
        (item.get("variable"), item)
        for item in section["table"]
        if isinstance(item, dict)
    )
if not entries:
    entries.extend(
        (key, value)
        for key, value in section.items()
        if isinstance(value, dict) and _normalize_variable_name(key)
    )

{
  "discount_pct": 1969,
  "channel": 3883,
  "total_cells": 5852
}

["discount_pct", "channel"]

if transformations.get("timestamps_standardized") is not True:
    return False
if transformations.get("schema_drift_columns_filled") != ["discount_pct", "channel"]:
    return False

I was able to fix some of the tasks by minimally modifying the input on the images while others had to be kept out.

Answer Leakage and network issues

ALE does not restrict network as such. It’s a generally online bench and one of the authors mentioned “we keep it online so agent can sometimes fix its dependencies”.

Some tasks require the agent to look up things online or query APIs and these often came with URLs mentioned in the allow-lists. A few tasks stated network restrictions in their prompts.

Examples:

1. physical_sciences/lenacapavir_sar_table2_extraction says:

   “Use the staged paper PDF as your source of truth.”
   “Do not rely on web search or external answer sources.”

2. engineering/chisel_verilog_alignment_seq_1 states:

   “The VM has no internet access during solve time.”

The metadata for network policy (which tasks run online or offline) was not public at the time. Upstream later published a list of 5 tasks that must run offline.

I did 2-3 runs with GPT-5.6 Sol (Medium) without network isolation and found a few tasks that looked up solutions online. These tasks are made to run with network isolation now.

My personal favourite examples of agents looking up stuff on the internet were:

1. computing_math/go_game_reconstruction_1

   The goal was to reconstruct a Go game from the evidence staged in the sandbox. Instead, the agent realised that the target may be an identifiable public professional game.

   The prompt mentioned “Do not use a Go engine or a web browser.” so the agent proceeded to use curl and download an archive of more than 90,000 professional SGF games, identified the exact historical game, and recovered its 168 moves.

   

   No Go engine or web browser? The agent used curl to find the game in a public archive instead.

2. health_medicine/wsi_tumor_localization_1

   The task asks for the center point of a tumor in a whole-slide pathology image. The agent started by doing the actual image analysis, zooming into suspicious tissue patches.

   The input had retained its public filename, tumor_001.tif, which probably motivated the agent to look it up online, so the agent searched for the corresponding annotation, tumor_001.xml. It tried GitHub, grep.app, Hugging Face, Google, Bing, DuckDuckGo, the original CAMELYON16 challenge website, and finally Kaggle.

Broken inputs and missing dependencies

Here are a couple of examples.

In computing_math/synthetic_causal_structure_inference, the 40 datasets were arranged in blocks of five, matching the order of the eight scenario families listed in the prompt. The first five were “observed confounder,” the next five “backdoor criterion,” and so on. One agent used SCENARIOS[index // 5] to recover the scenario from the dataset’s position. This also let it derive several related answers, together worth 70% of the rubric, without inferring them from the data.

In engineering/humanoid_wbc_policy_evaluation, the supplied environment depended on a specific MuJoCo development build, 3.7.0.dev886334046. Its download URL 404’ed. The task archive contained the project and its lockfile, but no copy of the required package, so even downloading the complete archive wasn’t enough to recreate the environment. I excluded the task because I couldn’t get the required runtime working from the supplied files.

Missing or corrupt reference data

For some tasks, the reference data was missing or corrupt. This manifested as bad ground truth data, references being questionable or in 1-2 cases as scientifically questionable.

Some of these were fixed upstream. The unresolved ones had to be excluded from the subset.

I reported these issues to the ALE authors on Discord and they were fast to respond. They fixed the reference data for a few tasks in the upstream code and published the list of offline-only tasks (PR 1, commit).

The mitigations mentioned above in the benchmark issues section led us to the formation of ALE-Gold. “Gold” just means tasks I believe are not bad and measure capability well. Nine tasks had to be excluded, leaving us with 96 tasks.

One detail worth mentioning is that I looked at the inputs and traces of problematic tasks and decided on a case-by-case basis what to include. In some cases, the tasks were badly designed or had some issue, but were still included because they seemed to do fine across multiple rollouts.

ALE-Gold evaluation results

I ran full evaluations for four models that supported both text and vision. I passed on Opus 5 because of its higher cost and GLM 5.3 because it didn’t support vision.

The ALE CLI taskset used for these evaluations is public on the Prime Environments Hub.

These results are based on the 96-task ALE-Gold subset, with one reported result per model per task. Sol and Luna used xhigh reasoning, while Kimi K3 and GLM 5.3 Flash used max reasoning. Sol, Luna, and GLM used Codex; Kimi used Kimi Code. Rollouts that ended with execution errors were retried, with up to three retries per task.

Aggregate score is the mean task score on a 0–1 scale, including partial credit, with missing scores counted as zero. Pass rate counts only recorded scores exactly equal to 1.0.

Kimi K3 and GLM 5.3 Flash had several rollouts ending mid-turn / partial tool call execution. For Kimi K3, it was an ACP server issue on their side whereas for GLM 5.3 Flash, the issue seemed to be more due to a skill issue with the model. These were retried separately.

Sol (xhigh) vs Luna (xhigh)

Sol scored higher overall, with a mean score of 63.59% against Luna’s 56.72%. It also had the higher pass rate: 32.29% versus 27.08%, corresponding to 31 and 26 full-score results. Across the 96 tasks, Sol scored higher on 24, Luna on 17, and 55 were tied.

Luna used 35% more completion tokens, including reasoning, and made about 32% more model calls. Its recorded model-call cost was 86% lower: $19.19 versus $139.61 for Sol.

The four examples below show different reasons behind Luna’s advantages, from matching a reference verdict to following a numerical specification.

Selected task comparisons

Aggregate scores hide substantial differences between tasks. The five examples below compare all four models across financial analysis, scientific computing, and software tasks. Some show a clear score advantage; others show how similar scores can come from different approaches.

Failure modes

The dominant failure pattern was a skill issue across all models, including Sol and Luna, but it was most pronounced in GLM 5.3 Flash. Models sometimes used an incorrect calculation or implemented the required procedure incorrectly. They also replaced difficult parts of a workflow with simpler approximations without establishing that those approximations met the requirements. The resulting outputs could look plausible and pass basic checks while still drifting away from the requested method.

Verification could repeat the same assumptions as the implementation. Models checked that files existed, outputs had the right shape, or rerunning the code produced the same result, but these checks did not necessarily establish correctness.

In other cases, checks exposed a problem but the model submitted the result anyway. Luna noticed questionable masks in the cell-tracking task but left them unrepaired. GLM produced a loadable checkpoint in checkpoint consolidation despite large numerical errors. Detecting a problem did not always lead to correcting the final output. I believe this often happens when the model is not confident about what it is doing.

Models sometimes produced outputs that did not match the schema or format expected by the scorer, losing partial credit. These mismatches involved details such as column ordering, field names, enum/schema, or file structure.

Model-specific failure modes

More model-specific failure patterns also stood out.

Sol has a tendency to attempt an approximate version of a task when it finds the work difficult. It also sometimes changed its approach based on what it thought the evaluator expected, even when that meant departing from the task’s explicit requirements. Output mismatches with the scorer’s expected schema or format also led to lost credit, although it was difficult to determine their full extent because the direct causes of some score deductions remained unclear.

Luna persisted with weak approaches and sometimes submitted results despite checks exposing unresolved problems. In some tasks, it focused on satisfying the evaluator’s checks without adequately establishing that the underlying work met the requirements.

Kimi K3 substituted shortcuts for the requested workflow without adequately checking that they met the original requirements.

GLM 5.3 Flash often made implementation errors or used the wrong formulas. My interpretation is that these mistakes reflected the capability and knowledge limitations of a smaller model.

Performance by domain

Across the five domains with at least 10 tasks, the four-model average is highest in computing and mathematics, followed by business and finance, then physical sciences. Sol leads in business and finance, physical sciences, life sciences, and health and medicine; Luna leads in computing and mathematics. Sol’s advantages are larger in health and medicine, life sciences, and physical sciences, while they both wiped the floor in computing and mathematics. Among the smaller domains, Luna leads legal, while Kimi K3 leads education and information and narrowly leads engineering. GLM has individual task wins but no outright domain-mean lead. These smaller domains contain between one and five tasks, so their averages are sensitive to individual results.

Engineering has room for improvement

Among domains with at least five tasks, engineering has the lowest scores across all four models. Their mean scores are tightly clustered around 0.21. All four receive full credit on robot-description reconstruction, very little credit on power-feeder reliability, and zero on low-thrust trajectory design, building model-predictive control, and urban traffic calibration.

The traces show substantive method and implementation problems. In low-thrust trajectory design, Sol and Luna generated real trajectories but replaced the required state/costate shooting method with other approaches while still reporting successful shooting. In building control, Luna substituted thermostat control for the required discrete airflow actions, while GLM substituted scheduled actions for the requested model-predictive policy.

However, these scores also reflect problems with the tasks and their evaluation. The low-thrust task contains conflicting thrust, duration, and velocity-change requirements. The building-control scorer imposes output conventions that are not fully specified in the instructions. Urban traffic calibration awards credit only when all 12 checks pass, so a zero can hide substantial progress on the public calibration targets. Engineering offers room for improvement in both model performance and evaluation quality.

Conclusion

Once you start looking into benchmarks, you will realise that they are not perfect. Often, they are not measuring what they claim to measure, are fully saturated, and/or are unable to differentiate between the capabilities of models. This post was my attempt to highlight some of these issues.

Terminal-Bench 2.1 is a good example: it is no longer very useful for distinguishing the capabilities of leading models, while TB 4.0 still reveals substantial differences. SWE-2 scored 92.8% on TB 2.1, ahead of the other models in the comparison, but only 27.3% on TB 4.0, compared with 55.8% for Fable 5.1 and 57.9% for GPT-6 Astra.

lmao lower TerminalBench 4.0 and DeepSWE than DeepSeek V4.1

— Teortaxes (@teortaxesTex) September 10, 2026

That said, Agents’ Last Exam is not yet saturated (and note that we looked at only its public Linux CLI subset). I expect many domains of ALE to be rapidly hillclimbed in the next six months.

This exploration was an overall great learning experience for me. I hope you got to learn something new too! Thanks for reading! Please share/upvote if you liked it.

Acknowledgements

Xeophon (Florian Brand), for being my peer, discussing ideas with me, and guiding me whenever I was stuck.

Snimu (Sebastian Muller), for coordinating the residency and providing the compute and guidance whenever I requested them.

Thanks to Prime Intellect for the Prime Residency program. Thanks also to the Prime Intellect team for building such great infrastructure. This work would not have been possible without their infrastructure.

#AI #featured