sankalp's blog

SWE-bench Multimodal v2 is not multimodal

Contents

Intro

I was using SWE-bench Multimodal v2 to establish baselines for a project at the Prime Intellect residency. I found out that GPT-6 Luna (xhigh) performed similarly on the benchmark whether I used normal images or OCR-based output or no images at all. This felt unexpected for a multimodal benchmark. I wondered: does this mean the images are optional?

After looking into the data, we found that most of the information the models needed was already in the issue text or derivable from the code. I also ran Sonnet 5.5 on a subset to check whether this was specific to Luna. Its score barely changed when I removed the images. In short,

What changes when the images disappear?

with imagesno imageschange, points
SWE-bench Multimodal v2 Β· resolved
GPT-6 Luna xhigh
480 tasks
36.9%
37.1%
+0.2
Sonnet 5.5 max
60 tasks
35.0%
40.0%
+5.0
MMMU-Pro Β· accuracy
GPT-6 Luna xhigh
1,730 questions
79.7%
36.9%
βˆ’42.8
Sonnet 5.5 max
120 questions
80.8%
44.2%
βˆ’36.6

Scores barely move on SWE-bench Multimodal, while accuracy falls sharply on MMMU-Pro. Each pair uses the same tasks within that experiment.

Vals AI released a SWE-bench Multimodal v2 leaderboard on October 14, so this felt like a good time to write this blog up and share our findings.

Methodology

SWE Multimodal v2 Benchmark details

The benchmark has 480 tasks built from GitHub issues in 11 JavaScript repos (Carbon alone accounts for 133), most with screenshots. 39 of the 480 have no image in the issue text at all. It's worth noting that even these tasks can involve images, since the agent may take screenshots to check its work or open image files already in the repo.

Most tasks are small fixes to UI components, map rendering, syntax highlighting or Lighthouse audits, things like a missing aria-label, a misaligned icon or a keyword highlighted in the wrong colour. They are fairly focused and short horizon. Luna xhigh spent a median of 5.5 minutes on each. Vals AI reports 5min 14 sec median time for Luna and 24m+ for Sonnet 5.5 at max reasoning. I find it interesting that this benchmark is not saturated given the nature of tasks.

Tasks per repo (click to expand)
RepoTasks
carbon-design-system/carbon133
openlayers/openlayers77
GoogleChrome/lighthouse54
bpmn-io/bpmn-js53
highlightjs/highlight.js39
PrismJS/prism38
alibaba-fusion/next36
grommet/grommet20
prettier/prettier13
eslint/eslint11
scratchfoundation/scratch-gui6

Experiment details

I evaluated two models on SWE-bench Multimodal v2 and MMMU-Pro:

Model SWE-bench Multimodal v2 MMMU-Pro Input conditions
GPT-6 Luna (xhigh) All 480 tasks All 1,730 questions Images, OCR, no images
Sonnet 5.5 (max) 60-task subset 120-question subset Images, no images

For SWE-bench Multimodal, rollouts ran with full network isolation, and grading took place in a separate sandbox. Issue images were downloaded beforehand so they could be provided without network access.

Removing Git history

In the traces of an initial run, we caught agents:

As you can see, sometimes the agents take 'learn from the codebase' too literally. This is a known SWE-bench reward hack, documented about a year ago, and we were surprised to still see this in the benchmark images.

Checking out an older commit takes the files back in time, but it doesn't erase newer commits from .git. So the agent may start with the buggy code while the actual upstream fix is still sitting in the same repository. It can go on an archaeological expedition through Git refs or stored objects, find the fix, and copy it instead of solving the issue.

To prevent this, we rebuilt the benchmark images to keep only the task's base commit and the objects needed to reconstruct its files. We then copied the cleaned filesystem into a fresh image so the removed history couldn't survive in older image layers.

GPT-6 Luna xhigh runs

I was using GPT-6 Luna xhigh for ablations. There were three full runs on all 480 tasks, one per condition, on verifiers with the Codex harness.

The three conditions were native image input, a custom OCR-based representation of each image and no images mode.

This is possible to do in verifiers because of the interception server. It sits between the harness and the model API, letting us rewrite messages before the model sees them. I used it to replace images in both user messages and tool outputs with OCR text or placeholders.

In OCR mode, each image is replaced by the text that OCR reads from it, with every word placed on a character grid at its position in the image and tagged with its colour. A header gives the image size and main colours, and a short list underneath describes shapes that are not text, such as icons, outlines and lines.

The GitHub issue goes into the Codex harness; an interceptor rewrites the images in every model request as the native image, OCR text or a placeholder before it reaches GPT-6 Luna

The interceptor rewrites every image in each model request, in the issue and in tool outputs alike, before it reaches the model.

Sample trace

Issue message of task lighthouse-4036, which has 2 screenshots, as the model received it in each of the three modes.

Swipe or scroll sideways, or jump to with images Β· with OCR Β· no images

With images1 / 3 β†’

lighthouse-4036 issue message in image mode: the issue text followed by the two screenshots

With OCR← 2 / 3 β†’

lighthouse-4036 issue message in OCR mode: the issue text followed by the OCR text of each screenshot

No images← 3 / 3

lighthouse-4036 issue message in no images mode: the issue text followed by a note and the markers Image 1 of 2 and Image 2 of 2

Full trace in the no images mode.

The full trace (click to expand) Β· resolved Β· 5 model calls Β· 7 commands Β· 60 s

Upstream PR

imagesThe 2 issue images the model never saw

issue image

issue image

thinkingModifying repository for user request

I need to modify the repository and inspect it further. The user mentioned that the rel=noopener audit should ignore mailto links, which seems to be related to a request issue. I should identify the specific project involved. First, I'll start with a preamble to explain what I'm doing and then proceed with the relevant commands. This way, the user will know I'm addressing their concerns directly!

messageI’ll locate the noopener audit and its tests, then make mailto: links exempt.

I’ll locate the noopener audit and its tests, then make mailto: links exempt.

run$ pwd && rg -n "noopener|rel.*audit|target=.*_blank" . --glob '!node_modules' --glob '!vendor'
pwd && rg -n "noopener|rel.*audit|target=.*_blank" . --glob '!node_modules' --glob '!vendor'
output Β· 65 lines
Chunk ID: 159d7e
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 2232
Output:
/testbed
./lighthouse-cli/test/smokehouse/dobetterweb/dbw-expectations.js:50:      'external-anchors-use-rel-noopener': {
./lighthouse-cli/test/smokehouse/dobetterweb/dbw-expectations.js:301:      'external-anchors-use-rel-noopener': {
./docs/understanding-results.md:187:| result | `Object` | The actual audit result, a copy of the audit object found in [audits](#audits). *NOTE: this property will likely be removed in upcoming releases; use the `id` property to lookup the result in the `audits` property.* |
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:22:<!-- we use a relative protocol url to test that `new URL(resource)` in the critical-request-chains.js audit can handle it -->
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:103:<template id="noopener-links-tmpl">
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:104:  <!-- FAIL - does not use rel="noopener" to open external link -->
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:105:  <a href="https://www.google.com/" target="_blank">external link</a>
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:106:  <!-- FAIL - does not use rel="noopener" and has no href attribute, giving an
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:108:  <a target="_blank">external link</a>
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:110:  <a href="https://www.google.com/" target="_blank" rel="noopener nofollow">external link that uses rel noopener and another unrelated rel attribute</a>
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:112:  <a href="./doesnotexist" target="_blank">internal link is ok</a>
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:294:  stampTemplate('noopener-links-tmpl', document.body);
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:374:  if (params.has('relnoopener')) {
./changelog.md:24:* audit: Ignore `href=javascript:.*` for `rel=noopener` audit ([#3574](https://github.com/googlechrome/lighthouse/pull/3574))
./changelog.md:298:Thanks to Katie Hempenius, Artur M, Kevin Farrugia, Piper Chester, Robin Drexler, and Mike Deverell. Due to the effort of these developers we now have an audit for checking for paste-blocked password fields, unified usage of toLocaleString(), a correct gulp recipe, a well-functioning chrome-launcher module, and improved clarity in the audit test descriptions.  **We truly appreciate all these improvements!**
./changelog.md:882:- **Audit:** Handle invalid URLs in `external-anchors-use-rel-noopener` audit - #1358
./changelog.md:927:- **Audit:** handle anchor tags with no href in 'external-anchors-use-rel-noopener' audit - #1238
./changelog.md:1030:- **Audit:** identify `noopener` when in list of link types in `rel` - #1035
./changelog.md:1089:- **Audit:** Add DBW audit for `rel=noopener` on external links - #912
./lighthouse-core/test/results/sample_v2.json:1198:      "helpText": "When a button doesn't have an accessible name, screen readers announce it as \"button\", making it unusable for users who rely on screen readers. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/button-name)."
./lighthouse-core/test/results/sample_v2.json:1533:      "helpText": "A value greater than 0 implies an explicit navigation ordering. Although technically valid, this often creates frustrating experiences for users who rely on assistive technologies. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/tabindex)."
./lighthouse-core/test/results/sample_v2.json:2135:    "external-anchors-use-rel-noopener": {
./lighthouse-core/test/results/sample_v2.json:2146:            "url": "<a href=\"https://www.google.com/\" target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:2152:            "url": "<a target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:2158:            "url": "<a href=\"./doesnotexist\" target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:2163:      "name": "external-anchors-use-rel-noopener",
./lighthouse-core/test/results/sample_v2.json:2165:      "description": "Opens external anchors using `rel=\"noopener\"`",
./lighthouse-core/test/results/sample_v2.json:2166:      "helpText": "Open new tabs using `rel=\"noopener\"` to improve performance and prevent security vulnerabilities. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/noopener).",
./lighthouse-core/test/results/sample_v2.json:5125:            "helpText": "When a button doesn't have an accessible name, screen readers announce it as \"button\", making it unusable for users who rely on screen readers. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/button-name)."
./lighthouse-core/test/results/sample_v2.json:5580:            "helpText": "A value greater than 0 implies an explicit navigation ordering. Although technically valid, this often creates frustrating experiences for users who rely on assistive technologies. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/tabindex)."
./lighthouse-core/test/results/sample_v2.json:6392:          "id": "external-anchors-use-rel-noopener",
./lighthouse-core/test/results/sample_v2.json:6405:                  "url": "<a href=\"https://www.google.com/\" target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:6411:                  "url": "<a target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:6417:                  "url": "<a href=\"./doesnotexist\" target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:6422:            "name": "external-anchors-use-rel-noopener",
./lighthouse-core/test/results/sample_v2.json:6424:            "description": "Opens external anchors using `rel=\"noopener\"`",
./lighthouse-core/test/results/sample_v2.json:6425:            "helpText": "Open new tabs using `rel=\"noopener\"` to improve performance and prevent security vulnerabilities. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/noopener).",
./lighthouse-core/test/report/v2/renderer/dom-test.js:82:          'Some <a rel="noopener" target="_blank" href="https://example.com/foo">link</a>. ' +
./lighthouse-core/test/report/v2/renderer/dom-test.js:83:          '<a rel="noopener" target="_blank" href="http://example.com/">Learn more</a>.');
./lighthouse-core/test/report/v2/renderer/dom-test.js:87:          '<a rel="noopener" target="_blank" href="https://example.com/foo">link</a>',
./lighthouse-core/test/report/v2/renderer/dom-test.js:93:          '<a rel="noopener" target="_blank" href="https://example.com/foo"> Link </a> ' +
./lighthouse-core/test/report/v2/renderer/dom-test.js:115:          'good. <a rel="noopener" target="_blank" href="https://dequeuniversity.com/rules/axe/2.2/td-headers-attr">Learn more</a>.');
./lighthouse-core/test/report/v2/renderer/details-renderer-test.js:199:      assert.equal(el.rel, 'noopener');
./lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js:9:  require('../../../audits/dobetterweb/external-anchors-use-rel-noopener.js');
./lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js:16:describe('External anchors use rel="noopener"', () => {
./lighthouse-core/report/v2/templates.html:324:      <div class="lh-metadata__results">Results for: <a href="" class="lh-metadata__url" target="_blank" rel="noopener"><!-- fill me --></a></div>
./lighthouse-core/report/v2/templates.html:386:      <a href="https://github.com/GoogleChrome/Lighthouse/issues" target="_blank" rel="noopener">File an issue</a>
./lighthouse-core/report/v2/renderer/dom.js:108:        a.rel = 'noopener';
./lighthouse-core/report/v2/renderer/details-renderer.js:107:    a.rel = 'noopener';
./lighthouse-core/gather/gatherers/dobetterweb/anchors-with-no-rel-noopener.js:19:      const selector = 'a[target="_blank"]:not([rel~="noopener"])';
./lighthouse-core/config/default.js:30:      'dobetterweb/anchors-with-no-rel-noopener',
./lighthouse-core/config/default.js:153:    'dobetterweb/external-anchors-use-rel-noopener',
./lighthouse-core/config/default.js:340:        {id: 'external-anchors-use-rel-noopener', weight: 1},
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:17:      name: 'external-anchors-use-rel-noopener',
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:18:      description: 'Opens external anchors using `rel="noopener"`',
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:19:      failureDescription: 'Does not open external anchors using `rel="noopener"`',
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:20:      helpText: 'Open new tabs using `rel="noopener"` to improve performance and ' +
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:22:          '[Learn more](https://developers.google.com/web/tools/lighthouse/audits/noopener).',
thinkingInspecting instructions and files

I need to take a closer look at the instructions and the files available. I'm thinking maybe there are root AGENTS present in the current working directory, but I’m not sure. The developer mentioned that the root AGENT is included, but I can’t seem to find anything. So, I guess I’ll need to use ls or find to search for any applicable files or indications related to the AGENTS. Let's see what I can uncover!

run$ rg --files -g 'AGENTS.md' -g '!node_modules' -g '!vendor' /testbed / 2>/dev/null | head -80
rg --files -g 'AGENTS.md' -g '!node_modules' -g '!vendor' /testbed / 2>/dev/null | head -80
output Β· 6 lines
Chunk ID: 065d07
Wall time: 0.0480 seconds
Process exited with code 0
Original token count: 0
Output:
run$ sed -n '1,240p' lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js && printf '\n--- TEST ---\n' && sed -n '1,280p' lig…
sed -n '1,240p' lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js && printf '\n--- TEST ---\n' && sed -n '1,280p' lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
output Β· 161 lines
Chunk ID: 338291
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 1478
Output:
/**
 * @license Copyright 2016 Google Inc. All Rights Reserved.
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
 * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 */
'use strict';
​
const URL = require('../../lib/url-shim');
const Audit = require('../audit');
​
class ExternalAnchorsUseRelNoopenerAudit extends Audit {
  /**
   * @return {!AuditMeta}
   */
  static get meta() {
    return {
      name: 'external-anchors-use-rel-noopener',
      description: 'Opens external anchors using `rel="noopener"`',
      failureDescription: 'Does not open external anchors using `rel="noopener"`',
      helpText: 'Open new tabs using `rel="noopener"` to improve performance and ' +
          'prevent security vulnerabilities. ' +
          '[Learn more](https://developers.google.com/web/tools/lighthouse/audits/noopener).',
      requiredArtifacts: ['URL', 'AnchorsWithNoRelNoopener'],
    };
  }
​
  /**
   * @param {!Artifacts} artifacts
   * @return {!AuditResult}
   */
  static audit(artifacts) {
    let debugString;
    const pageHost = new URL(artifacts.URL.finalUrl).host;
    // Filter usages to exclude anchors that are same origin
    // TODO: better extendedInfo for anchors with no href attribute:
    // https://github.com/GoogleChrome/lighthouse/issues/1233
    // https://github.com/GoogleChrome/lighthouse/issues/1345
    const failingAnchors = artifacts.AnchorsWithNoRelNoopener
      .filter(anchor => {
        try {
          return new URL(anchor.href).host !== pageHost;
        } catch (err) {
          debugString = 'Lighthouse was unable to determine the destination ' +
              'of some anchor tags. If they are not used as hyperlinks, ' +
              'consider removing the _blank target.';
          return true;
        }
      })
      .filter(anchor => {
        // Ignore href's that are not real links
        return !anchor.href || !anchor.href.toLowerCase().startsWith('javascript:');
      })
      .map(anchor => {
        return {
          href: anchor.href || 'Unknown',
          target: anchor.target || '',
          rel: anchor.rel || '',
          url: '<a' +
              (anchor.href ? ` href="${anchor.href}"` : '') +
              (anchor.target ? ` target="${anchor.target}"` : '') +
              (anchor.rel ? ` rel="${anchor.rel}"` : '') + '>',
        };
      });
​
    const headings = [
      {key: 'href', itemType: 'url', text: 'URL'},
      {key: 'target', itemType: 'text', text: 'Target'},
      {key: 'rel', itemType: 'text', text: 'Rel'},
    ];
​
    const details = Audit.makeTableDetails(headings, failingAnchors);
​
    return {
      rawValue: failingAnchors.length === 0,
      extendedInfo: {
        value: failingAnchors,
      },
      details,
      debugString,
    };
  }
}
​
module.exports = ExternalAnchorsUseRelNoopenerAudit;
​
--- TEST ---
/**
 * @license Copyright 2016 Google Inc. All Rights Reserved.
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
 * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 */
'use strict';
​
const ExternalAnchorsAudit =
  require('../../../audits/dobetterweb/external-anchors-use-rel-noopener.js');
const assert = require('assert');
​
const URL = 'https://google.com/test';
​
/* eslint-env mocha */
​
describe('External anchors use rel="noopener"', () => {
  it('passes when links are from same hosts as the page host', () => {
    const auditResult = ExternalAnchorsAudit.audit({
      AnchorsWithNoRelNoopener: [
        {href: 'https://google.com/test'},
        {href: 'https://google.com/test1'},
      ],
      URL: {finalUrl: URL},
    });
    assert.equal(auditResult.rawValue, true);
    assert.equal(auditResult.details.items.length, 0);
    assert.equal(auditResult.details.items.length, 0);
  });
​
  it('fails when links are from different hosts than the page host', () => {
    const auditResult = ExternalAnchorsAudit.audit({
      AnchorsWithNoRelNoopener: [
        {href: 'https://example.com/test'},
        {href: 'https://example.com/test1'},
      ],
      URL: {finalUrl: URL},
    });
    assert.equal(auditResult.rawValue, false);
    assert.equal(auditResult.details.items.length, 2);
    assert.equal(auditResult.details.items.length, 2);
  });
​
  it('handles links with no href attribute', () => {
    const auditResult = ExternalAnchorsAudit.audit({
      AnchorsWithNoRelNoopener: [
        {href: ''},
        {href: 'http://'},
        {href: 'http:'},
      ],
      URL: {finalUrl: URL},
    });
    assert.equal(auditResult.rawValue, false);
    assert.equal(auditResult.details.items.length, 3);
    assert.equal(auditResult.details.items.length, 3);
    assert.ok(auditResult.debugString, 'includes debugString');
  });
​
  it('does not fail for links with javascript in href attribute', () => {
    const auditResult = ExternalAnchorsAudit.audit({
      AnchorsWithNoRelNoopener: [
        {href: 'javascript:void(0)'},
        {href: 'JAVASCRIPT:void(0)'},
      ],
      URL: {finalUrl: URL},
    });
    assert.equal(auditResult.rawValue, true);
    assert.equal(auditResult.details.items.length, 0);
  });
});
run$ sed -n '90,125p' lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html && cat package.json | head -100 && git status --short
sed -n '90,125p' lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html && cat package.json | head -100 && git status --short
output Β· 142 lines
Chunk ID: 6b5fe4
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 1393
Output:
  </style>
</template>
​
</head>
<body>
​
<div>
  <h2>Do better web tester page</h2>
  <span>Hi there!</span>
</div>
​
<section id="touchmove-section">touchmove section</section>
​
<template id="noopener-links-tmpl">
  <!-- FAIL - does not use rel="noopener" to open external link -->
  <a href="https://www.google.com/" target="_blank">external link</a>
  <!-- FAIL - does not use rel="noopener" and has no href attribute, giving an
    href value of '' when read, which will throw in a `new URL('')` constructor -->
  <a target="_blank">external link</a>
  <!-- PASS -->
  <a href="https://www.google.com/" target="_blank" rel="noopener nofollow">external link that uses rel noopener and another unrelated rel attribute</a>
  <!-- PASS -->
  <a href="./doesnotexist" target="_blank">internal link is ok</a>
</template>
​
<template id="password-inputs-can-be-pasted-into">
  <!-- FAIL - calls preventDefault on event -->
  <input type="password" onpaste="event.preventDefault();" />
  <!-- PASS -->
  <input type="password" />
  <!-- FAIL - returns false in event handler -->
  <input type="password" onpaste="return false;" />
​
</template>
​
<!-- FAIL(image-aspect-ratio): image is naturally 480x318 -->
{
  "name": "lighthouse",
  "version": "2.6.0",
  "description": "Lighthouse",
  "main": "./lighthouse-core/index.js",
  "bin": {
    "lighthouse": "./lighthouse-cli/index.js",
    "chrome-debug": "./chrome-launcher/manual-chrome-launcher.js"
  },
  "engines": {
    "node": ">=6"
  },
  "scripts": {
    "install-all": "npm-run-posix-or-windows install-all:task",
    "install-all:task": "yarn & yarn install-extension & yarn install-viewer & wait",
    "install-all:task:windows": "yarn && yarn install-extension && yarn install-viewer",
    "install-extension": "cd ./lighthouse-extension && yarn install",
    "install-viewer": "cd ./lighthouse-viewer && yarn install",
    "build-all": "npm-run-posix-or-windows build-all:task",
    "build-all:task": "yarn build-extension & yarn build-viewer & wait",
    "build-all:task:windows": "yarn build-extension && yarn build-viewer",
    "build-extension": "cd ./lighthouse-extension && yarn build",
    "build-viewer": "cd ./lighthouse-viewer && yarn build",
    "clean": "rimraf *.report.html *.report.dom.html *.report.json *.screenshots.html *.devtoolslog.json *.trace.json || true",
    "lint": "[ \"$CI\" = true ] && eslint --quiet -f codeframe . || eslint .",
    "smoke": "bash lighthouse-cli/test/smokehouse/run-all-tests.sh",
    "coverage": "istanbul cover -x \"**/third_party/**\" _mocha -- $(find */test -name '*-test.js') --timeout 10000 --reporter progress --report lcovonly",
    "coveralls": "yarn coverage && cat ./coverage/lcov.info | coveralls && ./node_modules/codecov/bin/codecov",
    "start": "node ./lighthouse-cli/index.js",
    "test": "yarn lint --quiet && yarn unit && yarn type-check && yarn closure",
    "unit-core": "bash lighthouse-core/scripts/run-mocha.sh --core",
    "unit-cli": "bash lighthouse-core/scripts/run-mocha.sh --cli",
    "unit-viewer": "bash lighthouse-core/scripts/run-mocha.sh --viewer",
    "unit": "bash lighthouse-core/scripts/run-mocha.sh --default",
    "core-unit": "yarn unit-core",
    "cli-unit": "yarn unit-cli",
    "viewer-unit": "yarn unit-viewer",
    "closure": "cd lighthouse-core && node closure/closure-type-checking.js",
    "devtools": "bash lighthouse-core/scripts/roll-to-devtools.sh",
    "compile-devtools": "bash lighthouse-core/scripts/compile-against-devtools.sh",
    "watch": "bash lighthouse-core/scripts/run-mocha.sh --watch",
    "chrome": "node chrome-launcher/manual-chrome-launcher.js",
    "fast": "yarn start -- --disable-device-emulation --disable-cpu-throttling --disable-network-throttling",
    "smokehouse": "node lighthouse-cli/test/smokehouse/smokehouse.js",
    "deploy-viewer": "cd lighthouse-viewer && gulp deploy",
    "bundlesize": "bundlesize",
    "plots-smoke": "bash plots/test/smoke.sh",
    "changelog": "conventional-changelog --config ./build/changelog-generator/index.js --infile changelog.md --same-file",
    "type-check": "tsc -p ."
  },
  "devDependencies": {
    "@types/configstore": "^2.1.1",
    "@types/inquirer": "^0.0.35",
    "@types/node": "*",
    "@types/opn": "^3.0.28",
    "@types/update-notifier": "^1.0.2",
    "@types/yargs": "^8.0.2",
    "babel-core": "^6.16.0",
    "bundlesize": "^0.14.4",
    "codecov": "^2.2.0",
    "commitizen": "^2.9.6",
    "conventional-changelog-cli": "^1.3.4",
    "coveralls": "^2.11.9",
    "cz-customizable": "^5.2.0",
    "eslint": "^4.8.0",
    "eslint-config-google": "^0.9.1",
    "google-closure-compiler": "^20170521.0.0",
    "gulp": "^3.9.1",
    "gulp-replace": "^0.5.4",
    "gulp-util": "^3.0.7",
    "istanbul": "^0.4.3",
    "jsdom": "^9.12.0",
    "mocha": "^3.2.0",
    "npm-run-posix-or-windows": "^2.0.2",
    "typescript": "^2.6.1",
    "zone.js": "^0.7.3"
  },
  "dependencies": {
    "axe-core": "2.4.1",
    "chrome-devtools-frontend": "1.0.422034",
    "chrome-launcher": "0.8.1",
    "configstore": "^3.1.1",
    "devtools-timeline-model": "1.1.6",
    "http-link-header": "^0.8.0",
    "inquirer": "^3.3.0",
    "jpeg-js": "0.1.2",
    "js-library-detector": "^4.0.0",
    "json-stringify-safe": "5.0.1",
    "lighthouse-logger": "^1.0.0",
    "metaviewport-parser": "0.1.0",
    "mkdirp": "0.5.1",
    "opn": "4.0.2",
    "parse-cache-control": "1.0.1",
    "raven": "^2.2.1",
    "rimraf": "^2.6.1",
    "semver": "^5.3.0",
    "speedline": "1.3.0",
    "update-notifier": "^2.1.0",
    "whatwg-url": "^6.3.0",
    "ws": "3.3.2",
thinkingRefining URL Filter Logic

I need to update the audit implementation to specifically exclude mailto links. The current filter ordering is causing issues since it's checking new URL mailto cases, and I need a way to parse this correctly. I should ensure JavaScript handling is included too. Maybe I can separate this with conditions to ignore actual links while still ensuring that mailto links aren't parsed incorrectly. Overall, I'm focused on simplifying the logic to avoid unnecessary URL parsing, while still being thorough in the audit checks.Updating Mailto Protocol Handling

It seems the new URL for mailto likely returns an empty host, which isn’t producing an exception. I should implement an extension in the predicate and set up a unit test that checks mixed-case mailto links, since that’s likely a requirement. There might be a need to create separate tests for mailto specifically. Also, I should consider adding a changelog entry for clarity and check if existing tests cover this. Focused changes are the goal, so I'll aim for simplicity while ensuring thorough testing.

messageThe audit currently exempts javascript: only; I’ll add a case-insensitive mailto: exemption and regression test.

The audit currently exempts javascript: only; I’ll add a case-insensitive mailto: exemption and regression test.

run$ apply_patch <<'PATCH' *** Begin Patch *** Update File: lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js @@ }) .filte…
apply_patch <<'PATCH'
*** Begin Patch
*** Update File: lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
@@
       })
       .filter(anchor => {
-        // Ignore href's that are not real links
-        return !anchor.href || !anchor.href.toLowerCase().startsWith('javascript:');
+        // Ignore href's that are not real links or do not navigate to a page.
+        const href = (anchor.href || '').toLowerCase();
+        return !anchor.href || (!href.startsWith('javascript:') && !href.startsWith('mailto:'));
       })
*** Update File: lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
@@
   it('does not fail for links with javascript in href attribute', () => {
     const auditResult = ExternalAnchorsAudit.audit({
       AnchorsWithNoRelNoopener: [
         {href: 'javascript:void(0)'},
@@
     assert.equal(auditResult.rawValue, true);
     assert.equal(auditResult.details.items.length, 0);
   });
+
+  it('does not fail for links with mailto in href attribute', () => {
+    const auditResult = ExternalAnchorsAudit.audit({
+      AnchorsWithNoRelNoopener: [
+        {href: 'mailto:sales@example.com'},
+        {href: 'MAILTO:support@example.com'},
+      ],
+      URL: {finalUrl: URL},
+    });
+    assert.equal(auditResult.rawValue, true);
+    assert.equal(auditResult.details.items.length, 0);
+  });
 });
*** End Patch
PATCH
output Β· 9 lines
Wall time: 0.0000 seconds
Output:
Exit code: 0
Wall time: 0 seconds
Output:
Success. Updated the following files:
M lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
M lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
messagePatch is in; I’ll run the focused audit test and inspect the final diff.

Patch is in; I’ll run the focused audit test and inspect the final diff.

run$ ./node_modules/.bin/mocha lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js --timeout 10000
./node_modules/.bin/mocha lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js --timeout 10000
output Β· 18 lines
Chunk ID: 648f78
Wall time: 0.2552 seconds
Process exited with code 0
Original token count: 92
Output:
​
​
  External anchors use rel="noopener"
    βœ“ passes when links are from same hosts as the page host
    βœ“ fails when links are from different hosts than the page host
    βœ“ handles links with no href attribute
    βœ“ does not fail for links with javascript in href attribute
    βœ“ does not fail for links with mailto in href attribute
​
​
  5 passing (20ms)
run$ git diff --check && git diff -- lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js lighthouse-core/test/audits/dobette…
git diff --check && git diff -- lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js && git status --short
output Β· 45 lines
Chunk ID: 59373a
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 510
Output:
diff --git a/lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js b/lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
index 296f94d..21b1147 100644
--- a/lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
+++ b/lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
@@ -47,8 +47,9 @@ class ExternalAnchorsUseRelNoopenerAudit extends Audit {
         }
       })
       .filter(anchor => {
-        // Ignore href's that are not real links
-        return !anchor.href || !anchor.href.toLowerCase().startsWith('javascript:');
+        // Ignore href's that are not real links or do not navigate to a page.
+        const href = (anchor.href || '').toLowerCase();
+        return !anchor.href || (!href.startsWith('javascript:') && !href.startsWith('mailto:'));
       })
       .map(anchor => {
         return {
diff --git a/lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js b/lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
index c89aab4..ea8d777 100644
--- a/lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
+++ b/lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
@@ -66,4 +66,16 @@ describe('External anchors use rel="noopener"', () => {
     assert.equal(auditResult.rawValue, true);
     assert.equal(auditResult.details.items.length, 0);
   });
+
+  it('does not fail for links with mailto in href attribute', () => {
+    const auditResult = ExternalAnchorsAudit.audit({
+      AnchorsWithNoRelNoopener: [
+        {href: 'mailto:sales@example.com'},
+        {href: 'MAILTO:support@example.com'},
+      ],
+      URL: {finalUrl: URL},
+    });
+    assert.equal(auditResult.rawValue, true);
+    assert.equal(auditResult.details.items.length, 0);
+  });
 });
 M lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
 M lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
thinkingChecking unit test results

The unit test shows it can handle both lower and uppercase, and I have 5 validation tests passing, which is great! I don't see the need for any more tests at this point. However, I'm a bit concerned that the status output indicates only two modifications, but I'll keep an eye on that. Overall, it seems I can finalize with up to 10 lines.

messageUpdated the rel="noopener" audit to ignore mailto: links, case-insensitively, in lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:50. Ad…

Updated the rel="noopener" audit to ignore mailto: links, case-insensitively, in lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:50. Added lowercase and uppercase regression coverage in lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js:69.

Validation: all 5 focused audit tests pass.

The model didn't need the screenshots here (a Lighthouse audit flagging the mailto link, and a broken link on Android). The issue title already says to ignore mailto links, and the audit already skipped javascript: links, so it just added mailto: next to that.


Sonnet 5.5 max runs

It was interesting to see that Claude models were ahead of all OpenAI models on the leaderboard, which made me pause and worry a bit. Maybe Luna just doesn't make good use of images, and a model that does would show a gap. I shared my concern with Florian and he suggested we could run Sonnet 5.5 as well.

Vals AI's SWE-bench Multimodal leaderboard. Claude Opus 5.5 70.21%, Claude Fable 5.1 61.25%, Claude Sonnet 5.5 58.33% and Claude Haiku 5.5 43.96% all score above GPT-6.1 Sol 43.54%, GPT-6 Astra 43.33% and GPT-6 Luna 37.50%.

Vals AI's SWE-bench Multimodal leaderboard as of October 20.

Sonnet 5.5 at max reasoning is slow and pricey, so I picked a 60-task subset. Those 60 tasks cost about $170, more than the whole 480-task Luna run (about $110, inference plus sandboxes).

Sonnet runs at max effort (adaptive thinking) in the Claude Code harness, on 60 randomly sampled tasks spread over all 11 repos. The same tasks were used for both runs, native images and no images, with one rollout per task. No OCR run this time.

The rest of the setup was the same as the Luna runs, except for the agent timeout limit, which was raised from 20 to 50 minutes since Vals AI reports Sonnet at max effort taking 24m+ per task.

Results

GPT-6 Luna xhigh (480 tasks)

All three runs resolved 175 to 178 of 480 tasks (36.5 to 37.1%). This is in line with the 37.5% that Vals AI's SWE-bench Multimodal leaderboard reports for GPT-6 Luna on max effort (as of October 2026), even though they run it in a different harness (mini-swe-agent).

Run Resolved Rate
Native images 177 / 480 36.9%
OCR text 175 / 480 36.5%
No images 178 / 480 37.1%

Pair Both resolved Only the first Only the second Both failed
Images vs OCR 163 14 12 291
Images vs no images 167 10 11 292
OCR vs no images 161 14 17 288

157 tasks were resolved in all three runs and 284 failed in all three, so 441 of 480 (92%) got the same result every time. The remaining 39 change outcomes, with gains and losses roughly balancing out (images vs no images is βˆ’0.2 points, 95% CI βˆ’2.1 to +1.7, McNemar p = 1.0).

Observations

Some observations by my trace police (Sonnet 5.5, sometimes Opus 5.5 reviewer agents) after going through the traces:

Sonnet 5.5 max (60-task subset)

Run Sonnet 5.5 max GPT-6 Luna xhigh on the same 60
Native images 21 / 60 (35.0%) 14 / 60 (23.3%)
OCR text not run 10 / 60 (16.7%)
No images 24 / 60 (40.0%) 14 / 60 (23.3%)

19 tasks were resolved in both Sonnet runs and 34 failed in both. 2 flipped towards images and 5 towards no images (βˆ’5 points, 95% CI roughly βˆ’15 to +5, McNemar p = 0.45). With only 60 tasks, this shows no sign of an image benefit but can't rule out a moderate one.

Median thinking per task was 69k tokens with images and 67k without them. The no-image runs took a few more tool calls (median 82 vs 67) and a bit longer (18 vs 14 minutes). In contrast, Sonnet used 3x the output tokens on MMMU-Pro when images were gone.

Observations

Some observations by my trace police after going through the traces:

Why did removing images make so little difference?

To check whether images were slipping through somewhere, I had the trace police inspect both models' runs and checked some samples myself. The audit found no issue images reaching either model in the no-image runs.

The reviewers also assessed whether each task needed its images. For each task, they examined the native-image trace alongside the issue images, the merged fix and the hidden tests. These ratings are the reviewers' judgments and not definitive labels.

Images were rarely required

GPT-6 Luna xhigh

In 403 of 480 tasks (84%) the issue text fully specifies the bug and the images only illustrate it. Resolution counts barely move across the three runs. The helpful category has nine resolutions in each run, and of the seven tasks rated required, only one was resolved, with native images.

Reviewer rating Tasks Resolved: images / OCR / omitted
not_needed: text fully specifies the task 403 167 / 166 / 169
helpful: text is enough, images remove ambiguity 70 9 / 9 / 9
required: needed information is only in the images 7 1 / 0 / 0

Where images mattered at all, it was the issue screenshots, never an image file from the repo.

Seven required tasks (click to expand)
TaskImages / OCR / omittedWhat only the images showWhat happened in the native run
carbon-12384R / F / FSpec: read-only steppers stay visible but greyedResolved. The only clear image win in 480 tasks
lighthouse-15092F / F / FWhole error redesign; issue text is the title onlyTests check the category gauge, which Luna left alone
lighthouse-9151F / F / FWhich gauge is broken (PWA badge)Patched the path shape; the cause was duplicate SVG ids
lighthouse-4301F / F / FGatherer names during the hang point to start-urlPatched driver listener setup instead of the gatherer
carbon-3139F / F / FThe two accessibility violationsRight fix, but undefined where snapshots expect null
carbon-4260F / F / FThe two accessibility violationsSnapshot expects an id scheme nobody specified
openlayers-14100F / F / FWMTS GetTile URLs with TIMERead it as a throttling question; no patch

Sonnet 5.5 max

The reviewers rated the images not needed in 53 of 60 tasks and required in none. Resolution is flat again. Without images Sonnet resolves 4 more of the not_needed tasks and 1 fewer of the helpful ones.

Reviewer rating Tasks Resolved: images / omitted
not_needed: text fully specifies the task 53 17 / 21
helpful: text is enough, images remove ambiguity 5 2 / 1
required: needed information is only in the images 0 –
no images in the issue 2 2 / 2

The reviewers of the no-image traces rated next-3724 as required and moved grommet-6438 and highlight.js-2969 up to helpful (both failed either way). The two audits agree on the other 57 tasks.

Five helpful tasks (click to expand)
TaskImages / omittedWhat only the images showWhat happened
next-3724R / FKeep the left search box and hide the right oneWithout the screenshot the agent made only searchProps per-panel, so the right box still rendered. The only clear image win in the subset
carbon-12384R / RSpec: read-only steppers stay visible but greyedResolved both ways. The test only checks that the steppers are disabled
carbon-14476F / FThe "Select all (N)" label and divider layoutHid the button once every row was selected; the test expects it to stay
carbon-4028F / FThe four accessibility violationsLeft ListBoxSelection alone; snapshots expect aria-label="Clear Selection"
carbon-4820F / FThe label-in-name and contrast violationsPatched only Dropdown; snapshots expect a new id scheme across five components

The hidden tests don't check what the images show

The audit also exposed a gap between what the screenshots specify and what the hidden tests reward. A lot of the Carbon issues come with detailed design spec screenshots (colours, spacing, icon placement), but the test that grades them is often just a snapshot of the component's prop types (carbon-13851, for example, comes with a full TimePicker error/warning spec, and the test only checks that two new props exist). The OpenLayers and bpmn-js issues have GIFs of the bug, while the tests check internal state like waypoints or function arguments.

carbon-13851 design spec: TimePicker error state with a red border, red error icon and red message, and a warning state with a yellow warning icon, annotated with 8px and 16px spacing
What the screenshot specifies
// PublicAPI-test.js.snap, TimePicker props
       "value": Object {
         "type": "string",
       },
+      "warning": Object {
+        "type": "bool",
+      },
+      "warningText": Object {
+        "type": "node",
+      },
Everything the grader checks

The test only checks that warning and warningText exist as props. The red border, the icons, the message colours and the 8px and 16px spacing are never checked.

When tasks failed, it was usually because the tests expected some detail that neither the text nor the images mention, like an exact prop name, null vs undefined, or something settled in the GitHub issue comments that the model never sees. The reviewers flagged this in all but 2 of the Luna failures they reviewed, and in 32 of Sonnet's 36 no-image failures. I haven't checked every case by hand.

For the images to matter, a test would have to check something you can only learn from them, like which component is broken or a value from the design spec.

The models may already know some fixes

Memorisation is another possibility. These tasks come from public GitHub issues and PRs from 2017–2024, so both models may already know some of the fixes and have little need for the screenshots. In fact, Sonnet tried to recall the upstream fix in 38 of 60 no-image runs.

Example tasks where images were not needed

These ten tasks were resolved with native images, OCR text and no images. You can read their full no-image traces in an interactive trace viewer. It lives on my other website, since Bear Blog can't run one.

Task What the images show What the text already says
lighthouse-4036 The noopener audit flagging mailto links; Chrome for Android opening a blank page Title: the audit "should ignore href="mailto:x@y.com" links". The repo already skips javascript: links the same way
prism-1602 Rendered YAML where "world" loses its colour when followed by # test "String highlighting does not work when a string has a comment on the same line", plus the repro - "world" # test
next-2984 Console TypeError "Cannot add property value, object is not extensible" CascaderSelect changes the props passed in; "the component should not change the user's values"
bpmn-js-1655 Diagram whose external labels are drawn in the pale fill colour "External labels should be set to the 'stroke' color, as seen in the embedded labels"
carbon-6675 Pagination footer reading "1 of 1 pages" Names the file, the lines and the rule: singular when max === 1 or totalPages === 1
eslint-14242 Terminal: exit code 1 without --quiet, exit code 0 with it "The exit code should be 1 when max-warnings is exceeded"
grommet-6282 DataTable sorted by Parent, rows with null Parent not grouped "Undefined properties should be treated as empty string"
highlight.js-2958 Output where <a:FAIL …> is plain text and the other tags are coloured Sample code labels each line OK or FAIL; title: "single letter namespace prefix not highlighted"
openlayers-14627 WebGL stack trace; debugger in fromEPSG4326 with Infinity inputs Repro code, the cause ("Math.tan will transform Infinity to NaN") and the expected behaviour
prettier-11884 Editor showing a lone comma inside [ ] Input, CLI options, the bad output and the expected output

MMMU-Pro as a control

Florian and I wanted to check that our setup really takes the image information away, and that Luna uses images when they matter, so I ran the same setup of native image input, customised OCR and placeholder images on MMMU-Pro, with the same model, reasoning effort and interceptor.

MMMU-Pro makes a good control because it isn't agentic. It's single-turn multiple-choice questions across college subjects, with no repo to read and no tools to fall back on. In its standard split, each question comes with a diagram, chart or photo that carries information the text does not restate, so a model that cannot see the image should actually lose points there.

GPT-6 Luna xhigh (1730 questions)

Looks like the images matter!

Run Accuracy Correct Output tokens / question
Native images 79.7% 1378 / 1730 1.7k
OCR text 56.3% 974 / 1730 4.1k
No images (placeholder) 36.9% 638 / 1730 2.1k

Every gap is large and paired (McNemar p < 1e-49 for all three pairs). Taking the images away costs 43 points here; on SWE-bench Multimodal it cost nothing. The OCR run also burns more than twice the output tokens of the native run (reasoning tokens alone: 4.0k vs 1.6k per question), which I interpret as the model trying to understand/reconstruct the figure from text.

The steepest drops come where the image carries structure the text leaves out: Art goes from 42/53 to 3/53 without images, Finance from 59/60 to 15/60, Accounting from 51/58 to 10/58. They are smallest where the text already carries the question: Literature 43/52 β†’ 38/52, Energy and Power 50/58 β†’ 38/58.

MMMU-Pro results by subject (click to expand)

Sorted by how much accuracy drops when the images are removed. Cell colour goes from red (low) to green (high). The biggest drops are highlighted.

SubjectNativeOCRNo imagesDrop (native β†’ none)
Art79%
42/53
34%
18/53
6%
3/53
βˆ’74 pts
Finance98%
59/60
88%
53/60
25%
15/60
βˆ’73 pts
Accounting88%
51/58
83%
48/58
17%
10/58
βˆ’71 pts
Design83%
50/60
40%
24/60
20%
12/60
βˆ’63 pts
Marketing95%
56/59
71%
42/59
34%
20/59
βˆ’61 pts
Computer Science77%
46/60
42%
25/60
20%
12/60
βˆ’57 pts
Public Health86%
50/58
81%
47/58
31%
18/58
βˆ’55 pts
Art Theory84%
46/55
49%
27/55
31%
17/55
βˆ’53 pts
Biology78%
46/59
42%
25/59
27%
16/59
βˆ’51 pts
Chemistry82%
49/60
47%
28/60
32%
19/60
βˆ’50 pts
Geography79%
41/52
48%
25/52
29%
15/52
βˆ’50 pts
Management72%
36/50
48%
24/50
22%
11/50
βˆ’50 pts
Materials82%
49/60
53%
32/60
32%
19/60
βˆ’50 pts
Economics90%
53/59
64%
38/59
41%
24/59
βˆ’49 pts
Physics85%
51/60
52%
31/60
42%
25/60
βˆ’43 pts
Architecture and Engineering83%
50/60
62%
37/60
43%
26/60
βˆ’40 pts
Electronics97%
58/60
63%
38/60
57%
34/60
βˆ’40 pts
Music63%
38/60
30%
18/60
25%
15/60
βˆ’38 pts
Basic Medical Science83%
43/52
40%
21/52
46%
24/52
βˆ’37 pts
Math87%
52/60
63%
38/60
53%
32/60
βˆ’33 pts
Psychology72%
43/60
48%
29/60
38%
23/60
βˆ’33 pts
Pharmacy79%
45/57
63%
36/57
47%
27/57
βˆ’32 pts
History70%
39/56
68%
38/56
39%
22/56
βˆ’30 pts
Clinical Medicine73%
43/59
46%
27/59
44%
26/59
βˆ’29 pts
Sociology74%
40/54
61%
33/54
46%
25/54
βˆ’28 pts
Agriculture57%
34/60
45%
27/60
33%
20/60
βˆ’23 pts
Mechanical Engineering71%
42/59
61%
36/59
49%
29/59
βˆ’22 pts
Energy and Power86%
50/58
71%
41/58
66%
38/58
βˆ’21 pts
Diagnostics and Laboratory Medicine55%
33/60
47%
28/60
38%
23/60
βˆ’17 pts
Literature83%
43/52
77%
40/52
73%
38/52
βˆ’10 pts

Sonnet 5.5 max (120 questions)

I also ran Sonnet 5.5 at max effort on 120 of these questions (4 random ones per subject), with and without images, in the same plain chat setup. Luna's numbers on the same 120 are on the right for comparison.

Run Sonnet 5.5 max Output tokens / question GPT-6 Luna xhigh on the same 120
Native images 80.8% (97 / 120) 6.8k 78.3% (94 / 120)
OCR text not run 53.3% (64 / 120)
No images 44.2% (53 / 120) 19.8k 37.5% (45 / 120)

Taking away the images costs Sonnet 37 points: 46 questions go from right to wrong and only 2 the other way (McNemar p < 1e-10). The same change cost it nothing on SWE-bench Multimodal. Two native questions failed because their images were over Anthropic's 5 MB limit; they count as wrong.

Without images, Sonnet uses about three times as many output tokens (6.8k to 19.8k). They say a picture is worth a thousand words. Apparently, removing one adds several thousand for Sonnet.

It says it can't see the image in 78 of the 120 questions and openly guesses in 92, often from the answer options themselves. Opus 5.5 reviewers trace 55 of the 67 misses in the no-image run to the missing image.

Conclusion

The experiments show that SWE-bench Multimodal v2 scores were largely insensitive to removing image inputs in our setups, while the same ablation cost both models around 40 points on MMMU-Pro. The audit points to two reasons. Screenshots often repeat what the issue text already says, and the hidden tests rarely check visual requirements.

Thanks for reading! If you found the post helpful, please like/retweet and share it with others who might find it useful.

Acknowledgements

Thanks to Florian for the suggestions along the way and encouraging me to blog about this. Also grateful to Prime Intellect for the infrastructure and the generous inference credits that made these runs possible.

#ai #featured