sankalp's blog

Removing images barely changed GPT-6 Luna’s SWE-bench Multimodal v2 score

Contents

Intro

I have been establishing baselines for a project at the Prime Intellect residency and we were using SWE-bench Multimodal v2 to run ablations. I found out that GPT 6 Luna xhigh performed on the benchmark similarly whether I used normal images or OCR or no images at all. This was a bit baffling at first but based on further investigation (and burning a lot of tokens), I concluded that most of the information required to solve the tasks was provided in the text part of the tasks or was derivable from code. In other words, SWE-bench Multimodal is mostly not testing the multimodality: the images carry needed information for only a handful of tasks. In this post, I will briefly go through the methodology and then the investigation results.

Methodology

The benchmark has 480 tasks built from GitHub issues in JavaScript repos, most with screenshots. 39 of the 480 have no image in the issue at all, so they get the same prompt in all three runs. 38 of these are OpenLayers tasks whose only images are the reference renders (expected.png) that the hidden rendering tests compare against, so the work can still be visual even though the model never sees an image. The last one is a Prism task whose image link is dead. The tasks are relatively focused and UI oriented: in the no-image run, Luna's median time per task was about 5.5 minutes.

I was using GPT 6 Luna xhigh for ablations. Cheapest model and higher rate limits. I did three full evaluations, one per condition, on verifiers with Codex harness with full network isolation. The issue images were downloaded in advance. We (Florian helped) also rebuilt the task Docker images with the git history objects removed from the ones provided by the benchmark because in an initial run, we found out that models were looking up “future” commits via the git object.

Evaluation runs

The three conditions were native image input, customised OCR and no images mode.

This is possible to do in verifiers because of the interception server which allows you to fetch the user message and tool related messages and replace stuff in them with what you want. I replaced the images with OCR or just placeholder text in the messages.

In OCR mode, each image is replaced by the text that OCR reads from it, with every word placed on a character grid at its position in the image and tagged with its colour. A header gives the image size and main colours, and a short list underneath describes shapes that are not text, such as icons, outlines and lines.

The GitHub issue goes into the Codex harness; an interceptor rewrites the images in every model request as the native image, OCR text or a placeholder before it reaches GPT 6 Luna

Sample trace

Below is a sample rollout trace of task lighthouse-4036 in the no images mode. The issue has 2 screenshots and the model gets the raw <img> placeholders along with a note that the images were removed. Here is the issue message as the model received it in each of the three modes.

lighthouse-4036 issue message in image mode: the issue text followed by the two screenshots

lighthouse-4036 issue message in OCR mode: the issue text followed by the OCR text of each screenshot

lighthouse-4036 issue message in no images mode: the issue text followed by a note and the markers Image 1 of 2 and Image 2 of 2

And here is the whole trace that followed in the no images mode: every reasoning summary, message and command with its output. Click a row to expand it.

The full trace (click to expand) · resolved · 5 model calls · 7 commands · 60 s

Upstream PR

imagesThe 2 issue images the model never saw

issue image

issue image

thinkingModifying repository for user request

I need to modify the repository and inspect it further. The user mentioned that the rel=noopener audit should ignore mailto links, which seems to be related to a request issue. I should identify the specific project involved. First, I'll start with a preamble to explain what I'm doing and then proceed with the relevant commands. This way, the user will know I'm addressing their concerns directly!

messageI’ll locate the noopener audit and its tests, then make mailto: links exempt.

I’ll locate the noopener audit and its tests, then make mailto: links exempt.

run$ pwd && rg -n "noopener|rel.*audit|target=.*_blank" . --glob '!node_modules' --glob '!vendor'
pwd && rg -n "noopener|rel.*audit|target=.*_blank" . --glob '!node_modules' --glob '!vendor'
output · 65 lines
Chunk ID: 159d7e
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 2232
Output:
/testbed
./lighthouse-cli/test/smokehouse/dobetterweb/dbw-expectations.js:50:      'external-anchors-use-rel-noopener': {
./lighthouse-cli/test/smokehouse/dobetterweb/dbw-expectations.js:301:      'external-anchors-use-rel-noopener': {
./docs/understanding-results.md:187:| result | `Object` | The actual audit result, a copy of the audit object found in [audits](#audits). *NOTE: this property will likely be removed in upcoming releases; use the `id` property to lookup the result in the `audits` property.* |
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:22:<!-- we use a relative protocol url to test that `new URL(resource)` in the critical-request-chains.js audit can handle it -->
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:103:<template id="noopener-links-tmpl">
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:104:  <!-- FAIL - does not use rel="noopener" to open external link -->
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:105:  <a href="https://www.google.com/" target="_blank">external link</a>
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:106:  <!-- FAIL - does not use rel="noopener" and has no href attribute, giving an
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:108:  <a target="_blank">external link</a>
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:110:  <a href="https://www.google.com/" target="_blank" rel="noopener nofollow">external link that uses rel noopener and another unrelated rel attribute</a>
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:112:  <a href="./doesnotexist" target="_blank">internal link is ok</a>
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:294:  stampTemplate('noopener-links-tmpl', document.body);
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:374:  if (params.has('relnoopener')) {
./changelog.md:24:* audit: Ignore `href=javascript:.*` for `rel=noopener` audit ([#3574](https://github.com/googlechrome/lighthouse/pull/3574))
./changelog.md:298:Thanks to Katie Hempenius, Artur M, Kevin Farrugia, Piper Chester, Robin Drexler, and Mike Deverell. Due to the effort of these developers we now have an audit for checking for paste-blocked password fields, unified usage of toLocaleString(), a correct gulp recipe, a well-functioning chrome-launcher module, and improved clarity in the audit test descriptions.  **We truly appreciate all these improvements!**
./changelog.md:882:- **Audit:** Handle invalid URLs in `external-anchors-use-rel-noopener` audit - #1358
./changelog.md:927:- **Audit:** handle anchor tags with no href in 'external-anchors-use-rel-noopener' audit - #1238
./changelog.md:1030:- **Audit:** identify `noopener` when in list of link types in `rel` - #1035
./changelog.md:1089:- **Audit:** Add DBW audit for `rel=noopener` on external links - #912
./lighthouse-core/test/results/sample_v2.json:1198:      "helpText": "When a button doesn't have an accessible name, screen readers announce it as \"button\", making it unusable for users who rely on screen readers. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/button-name)."
./lighthouse-core/test/results/sample_v2.json:1533:      "helpText": "A value greater than 0 implies an explicit navigation ordering. Although technically valid, this often creates frustrating experiences for users who rely on assistive technologies. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/tabindex)."
./lighthouse-core/test/results/sample_v2.json:2135:    "external-anchors-use-rel-noopener": {
./lighthouse-core/test/results/sample_v2.json:2146:            "url": "<a href=\"https://www.google.com/\" target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:2152:            "url": "<a target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:2158:            "url": "<a href=\"./doesnotexist\" target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:2163:      "name": "external-anchors-use-rel-noopener",
./lighthouse-core/test/results/sample_v2.json:2165:      "description": "Opens external anchors using `rel=\"noopener\"`",
./lighthouse-core/test/results/sample_v2.json:2166:      "helpText": "Open new tabs using `rel=\"noopener\"` to improve performance and prevent security vulnerabilities. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/noopener).",
./lighthouse-core/test/results/sample_v2.json:5125:            "helpText": "When a button doesn't have an accessible name, screen readers announce it as \"button\", making it unusable for users who rely on screen readers. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/button-name)."
./lighthouse-core/test/results/sample_v2.json:5580:            "helpText": "A value greater than 0 implies an explicit navigation ordering. Although technically valid, this often creates frustrating experiences for users who rely on assistive technologies. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/tabindex)."
./lighthouse-core/test/results/sample_v2.json:6392:          "id": "external-anchors-use-rel-noopener",
./lighthouse-core/test/results/sample_v2.json:6405:                  "url": "<a href=\"https://www.google.com/\" target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:6411:                  "url": "<a target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:6417:                  "url": "<a href=\"./doesnotexist\" target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:6422:            "name": "external-anchors-use-rel-noopener",
./lighthouse-core/test/results/sample_v2.json:6424:            "description": "Opens external anchors using `rel=\"noopener\"`",
./lighthouse-core/test/results/sample_v2.json:6425:            "helpText": "Open new tabs using `rel=\"noopener\"` to improve performance and prevent security vulnerabilities. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/noopener).",
./lighthouse-core/test/report/v2/renderer/dom-test.js:82:          'Some <a rel="noopener" target="_blank" href="https://example.com/foo">link</a>. ' +
./lighthouse-core/test/report/v2/renderer/dom-test.js:83:          '<a rel="noopener" target="_blank" href="http://example.com/">Learn more</a>.');
./lighthouse-core/test/report/v2/renderer/dom-test.js:87:          '<a rel="noopener" target="_blank" href="https://example.com/foo">link</a>',
./lighthouse-core/test/report/v2/renderer/dom-test.js:93:          '<a rel="noopener" target="_blank" href="https://example.com/foo"> Link </a> ' +
./lighthouse-core/test/report/v2/renderer/dom-test.js:115:          'good. <a rel="noopener" target="_blank" href="https://dequeuniversity.com/rules/axe/2.2/td-headers-attr">Learn more</a>.');
./lighthouse-core/test/report/v2/renderer/details-renderer-test.js:199:      assert.equal(el.rel, 'noopener');
./lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js:9:  require('../../../audits/dobetterweb/external-anchors-use-rel-noopener.js');
./lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js:16:describe('External anchors use rel="noopener"', () => {
./lighthouse-core/report/v2/templates.html:324:      <div class="lh-metadata__results">Results for: <a href="" class="lh-metadata__url" target="_blank" rel="noopener"><!-- fill me --></a></div>
./lighthouse-core/report/v2/templates.html:386:      <a href="https://github.com/GoogleChrome/Lighthouse/issues" target="_blank" rel="noopener">File an issue</a>
./lighthouse-core/report/v2/renderer/dom.js:108:        a.rel = 'noopener';
./lighthouse-core/report/v2/renderer/details-renderer.js:107:    a.rel = 'noopener';
./lighthouse-core/gather/gatherers/dobetterweb/anchors-with-no-rel-noopener.js:19:      const selector = 'a[target="_blank"]:not([rel~="noopener"])';
./lighthouse-core/config/default.js:30:      'dobetterweb/anchors-with-no-rel-noopener',
./lighthouse-core/config/default.js:153:    'dobetterweb/external-anchors-use-rel-noopener',
./lighthouse-core/config/default.js:340:        {id: 'external-anchors-use-rel-noopener', weight: 1},
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:17:      name: 'external-anchors-use-rel-noopener',
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:18:      description: 'Opens external anchors using `rel="noopener"`',
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:19:      failureDescription: 'Does not open external anchors using `rel="noopener"`',
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:20:      helpText: 'Open new tabs using `rel="noopener"` to improve performance and ' +
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:22:          '[Learn more](https://developers.google.com/web/tools/lighthouse/audits/noopener).',
thinkingInspecting instructions and files

I need to take a closer look at the instructions and the files available. I'm thinking maybe there are root AGENTS present in the current working directory, but I’m not sure. The developer mentioned that the root AGENT is included, but I can’t seem to find anything. So, I guess I’ll need to use ls or find to search for any applicable files or indications related to the AGENTS. Let's see what I can uncover!

run$ rg --files -g 'AGENTS.md' -g '!node_modules' -g '!vendor' /testbed / 2>/dev/null | head -80
rg --files -g 'AGENTS.md' -g '!node_modules' -g '!vendor' /testbed / 2>/dev/null | head -80
output · 6 lines
Chunk ID: 065d07
Wall time: 0.0480 seconds
Process exited with code 0
Original token count: 0
Output:
run$ sed -n '1,240p' lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js && printf '\n--- TEST ---\n' && sed -n '1,280p' lig…
sed -n '1,240p' lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js && printf '\n--- TEST ---\n' && sed -n '1,280p' lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
output · 161 lines
Chunk ID: 338291
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 1478
Output:
/**
 * @license Copyright 2016 Google Inc. All Rights Reserved.
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
 * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 */
'use strict';
​
const URL = require('../../lib/url-shim');
const Audit = require('../audit');
​
class ExternalAnchorsUseRelNoopenerAudit extends Audit {
  /**
   * @return {!AuditMeta}
   */
  static get meta() {
    return {
      name: 'external-anchors-use-rel-noopener',
      description: 'Opens external anchors using `rel="noopener"`',
      failureDescription: 'Does not open external anchors using `rel="noopener"`',
      helpText: 'Open new tabs using `rel="noopener"` to improve performance and ' +
          'prevent security vulnerabilities. ' +
          '[Learn more](https://developers.google.com/web/tools/lighthouse/audits/noopener).',
      requiredArtifacts: ['URL', 'AnchorsWithNoRelNoopener'],
    };
  }
​
  /**
   * @param {!Artifacts} artifacts
   * @return {!AuditResult}
   */
  static audit(artifacts) {
    let debugString;
    const pageHost = new URL(artifacts.URL.finalUrl).host;
    // Filter usages to exclude anchors that are same origin
    // TODO: better extendedInfo for anchors with no href attribute:
    // https://github.com/GoogleChrome/lighthouse/issues/1233
    // https://github.com/GoogleChrome/lighthouse/issues/1345
    const failingAnchors = artifacts.AnchorsWithNoRelNoopener
      .filter(anchor => {
        try {
          return new URL(anchor.href).host !== pageHost;
        } catch (err) {
          debugString = 'Lighthouse was unable to determine the destination ' +
              'of some anchor tags. If they are not used as hyperlinks, ' +
              'consider removing the _blank target.';
          return true;
        }
      })
      .filter(anchor => {
        // Ignore href's that are not real links
        return !anchor.href || !anchor.href.toLowerCase().startsWith('javascript:');
      })
      .map(anchor => {
        return {
          href: anchor.href || 'Unknown',
          target: anchor.target || '',
          rel: anchor.rel || '',
          url: '<a' +
              (anchor.href ? ` href="${anchor.href}"` : '') +
              (anchor.target ? ` target="${anchor.target}"` : '') +
              (anchor.rel ? ` rel="${anchor.rel}"` : '') + '>',
        };
      });
​
    const headings = [
      {key: 'href', itemType: 'url', text: 'URL'},
      {key: 'target', itemType: 'text', text: 'Target'},
      {key: 'rel', itemType: 'text', text: 'Rel'},
    ];
​
    const details = Audit.makeTableDetails(headings, failingAnchors);
​
    return {
      rawValue: failingAnchors.length === 0,
      extendedInfo: {
        value: failingAnchors,
      },
      details,
      debugString,
    };
  }
}
​
module.exports = ExternalAnchorsUseRelNoopenerAudit;
​
--- TEST ---
/**
 * @license Copyright 2016 Google Inc. All Rights Reserved.
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
 * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 */
'use strict';
​
const ExternalAnchorsAudit =
  require('../../../audits/dobetterweb/external-anchors-use-rel-noopener.js');
const assert = require('assert');
​
const URL = 'https://google.com/test';
​
/* eslint-env mocha */
​
describe('External anchors use rel="noopener"', () => {
  it('passes when links are from same hosts as the page host', () => {
    const auditResult = ExternalAnchorsAudit.audit({
      AnchorsWithNoRelNoopener: [
        {href: 'https://google.com/test'},
        {href: 'https://google.com/test1'},
      ],
      URL: {finalUrl: URL},
    });
    assert.equal(auditResult.rawValue, true);
    assert.equal(auditResult.details.items.length, 0);
    assert.equal(auditResult.details.items.length, 0);
  });
​
  it('fails when links are from different hosts than the page host', () => {
    const auditResult = ExternalAnchorsAudit.audit({
      AnchorsWithNoRelNoopener: [
        {href: 'https://example.com/test'},
        {href: 'https://example.com/test1'},
      ],
      URL: {finalUrl: URL},
    });
    assert.equal(auditResult.rawValue, false);
    assert.equal(auditResult.details.items.length, 2);
    assert.equal(auditResult.details.items.length, 2);
  });
​
  it('handles links with no href attribute', () => {
    const auditResult = ExternalAnchorsAudit.audit({
      AnchorsWithNoRelNoopener: [
        {href: ''},
        {href: 'http://'},
        {href: 'http:'},
      ],
      URL: {finalUrl: URL},
    });
    assert.equal(auditResult.rawValue, false);
    assert.equal(auditResult.details.items.length, 3);
    assert.equal(auditResult.details.items.length, 3);
    assert.ok(auditResult.debugString, 'includes debugString');
  });
​
  it('does not fail for links with javascript in href attribute', () => {
    const auditResult = ExternalAnchorsAudit.audit({
      AnchorsWithNoRelNoopener: [
        {href: 'javascript:void(0)'},
        {href: 'JAVASCRIPT:void(0)'},
      ],
      URL: {finalUrl: URL},
    });
    assert.equal(auditResult.rawValue, true);
    assert.equal(auditResult.details.items.length, 0);
  });
});
run$ sed -n '90,125p' lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html && cat package.json | head -100 && git status --short
sed -n '90,125p' lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html && cat package.json | head -100 && git status --short
output · 142 lines
Chunk ID: 6b5fe4
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 1393
Output:
  </style>
</template>
​
</head>
<body>
​
<div>
  <h2>Do better web tester page</h2>
  <span>Hi there!</span>
</div>
​
<section id="touchmove-section">touchmove section</section>
​
<template id="noopener-links-tmpl">
  <!-- FAIL - does not use rel="noopener" to open external link -->
  <a href="https://www.google.com/" target="_blank">external link</a>
  <!-- FAIL - does not use rel="noopener" and has no href attribute, giving an
    href value of '' when read, which will throw in a `new URL('')` constructor -->
  <a target="_blank">external link</a>
  <!-- PASS -->
  <a href="https://www.google.com/" target="_blank" rel="noopener nofollow">external link that uses rel noopener and another unrelated rel attribute</a>
  <!-- PASS -->
  <a href="./doesnotexist" target="_blank">internal link is ok</a>
</template>
​
<template id="password-inputs-can-be-pasted-into">
  <!-- FAIL - calls preventDefault on event -->
  <input type="password" onpaste="event.preventDefault();" />
  <!-- PASS -->
  <input type="password" />
  <!-- FAIL - returns false in event handler -->
  <input type="password" onpaste="return false;" />
​
</template>
​
<!-- FAIL(image-aspect-ratio): image is naturally 480x318 -->
{
  "name": "lighthouse",
  "version": "2.6.0",
  "description": "Lighthouse",
  "main": "./lighthouse-core/index.js",
  "bin": {
    "lighthouse": "./lighthouse-cli/index.js",
    "chrome-debug": "./chrome-launcher/manual-chrome-launcher.js"
  },
  "engines": {
    "node": ">=6"
  },
  "scripts": {
    "install-all": "npm-run-posix-or-windows install-all:task",
    "install-all:task": "yarn & yarn install-extension & yarn install-viewer & wait",
    "install-all:task:windows": "yarn && yarn install-extension && yarn install-viewer",
    "install-extension": "cd ./lighthouse-extension && yarn install",
    "install-viewer": "cd ./lighthouse-viewer && yarn install",
    "build-all": "npm-run-posix-or-windows build-all:task",
    "build-all:task": "yarn build-extension & yarn build-viewer & wait",
    "build-all:task:windows": "yarn build-extension && yarn build-viewer",
    "build-extension": "cd ./lighthouse-extension && yarn build",
    "build-viewer": "cd ./lighthouse-viewer && yarn build",
    "clean": "rimraf *.report.html *.report.dom.html *.report.json *.screenshots.html *.devtoolslog.json *.trace.json || true",
    "lint": "[ \"$CI\" = true ] && eslint --quiet -f codeframe . || eslint .",
    "smoke": "bash lighthouse-cli/test/smokehouse/run-all-tests.sh",
    "coverage": "istanbul cover -x \"**/third_party/**\" _mocha -- $(find */test -name '*-test.js') --timeout 10000 --reporter progress --report lcovonly",
    "coveralls": "yarn coverage && cat ./coverage/lcov.info | coveralls && ./node_modules/codecov/bin/codecov",
    "start": "node ./lighthouse-cli/index.js",
    "test": "yarn lint --quiet && yarn unit && yarn type-check && yarn closure",
    "unit-core": "bash lighthouse-core/scripts/run-mocha.sh --core",
    "unit-cli": "bash lighthouse-core/scripts/run-mocha.sh --cli",
    "unit-viewer": "bash lighthouse-core/scripts/run-mocha.sh --viewer",
    "unit": "bash lighthouse-core/scripts/run-mocha.sh --default",
    "core-unit": "yarn unit-core",
    "cli-unit": "yarn unit-cli",
    "viewer-unit": "yarn unit-viewer",
    "closure": "cd lighthouse-core && node closure/closure-type-checking.js",
    "devtools": "bash lighthouse-core/scripts/roll-to-devtools.sh",
    "compile-devtools": "bash lighthouse-core/scripts/compile-against-devtools.sh",
    "watch": "bash lighthouse-core/scripts/run-mocha.sh --watch",
    "chrome": "node chrome-launcher/manual-chrome-launcher.js",
    "fast": "yarn start -- --disable-device-emulation --disable-cpu-throttling --disable-network-throttling",
    "smokehouse": "node lighthouse-cli/test/smokehouse/smokehouse.js",
    "deploy-viewer": "cd lighthouse-viewer && gulp deploy",
    "bundlesize": "bundlesize",
    "plots-smoke": "bash plots/test/smoke.sh",
    "changelog": "conventional-changelog --config ./build/changelog-generator/index.js --infile changelog.md --same-file",
    "type-check": "tsc -p ."
  },
  "devDependencies": {
    "@types/configstore": "^2.1.1",
    "@types/inquirer": "^0.0.35",
    "@types/node": "*",
    "@types/opn": "^3.0.28",
    "@types/update-notifier": "^1.0.2",
    "@types/yargs": "^8.0.2",
    "babel-core": "^6.16.0",
    "bundlesize": "^0.14.4",
    "codecov": "^2.2.0",
    "commitizen": "^2.9.6",
    "conventional-changelog-cli": "^1.3.4",
    "coveralls": "^2.11.9",
    "cz-customizable": "^5.2.0",
    "eslint": "^4.8.0",
    "eslint-config-google": "^0.9.1",
    "google-closure-compiler": "^20170521.0.0",
    "gulp": "^3.9.1",
    "gulp-replace": "^0.5.4",
    "gulp-util": "^3.0.7",
    "istanbul": "^0.4.3",
    "jsdom": "^9.12.0",
    "mocha": "^3.2.0",
    "npm-run-posix-or-windows": "^2.0.2",
    "typescript": "^2.6.1",
    "zone.js": "^0.7.3"
  },
  "dependencies": {
    "axe-core": "2.4.1",
    "chrome-devtools-frontend": "1.0.422034",
    "chrome-launcher": "0.8.1",
    "configstore": "^3.1.1",
    "devtools-timeline-model": "1.1.6",
    "http-link-header": "^0.8.0",
    "inquirer": "^3.3.0",
    "jpeg-js": "0.1.2",
    "js-library-detector": "^4.0.0",
    "json-stringify-safe": "5.0.1",
    "lighthouse-logger": "^1.0.0",
    "metaviewport-parser": "0.1.0",
    "mkdirp": "0.5.1",
    "opn": "4.0.2",
    "parse-cache-control": "1.0.1",
    "raven": "^2.2.1",
    "rimraf": "^2.6.1",
    "semver": "^5.3.0",
    "speedline": "1.3.0",
    "update-notifier": "^2.1.0",
    "whatwg-url": "^6.3.0",
    "ws": "3.3.2",
thinkingRefining URL Filter Logic

I need to update the audit implementation to specifically exclude mailto links. The current filter ordering is causing issues since it's checking new URL mailto cases, and I need a way to parse this correctly. I should ensure JavaScript handling is included too. Maybe I can separate this with conditions to ignore actual links while still ensuring that mailto links aren't parsed incorrectly. Overall, I'm focused on simplifying the logic to avoid unnecessary URL parsing, while still being thorough in the audit checks.Updating Mailto Protocol Handling

It seems the new URL for mailto likely returns an empty host, which isn’t producing an exception. I should implement an extension in the predicate and set up a unit test that checks mixed-case mailto links, since that’s likely a requirement. There might be a need to create separate tests for mailto specifically. Also, I should consider adding a changelog entry for clarity and check if existing tests cover this. Focused changes are the goal, so I'll aim for simplicity while ensuring thorough testing.

messageThe audit currently exempts javascript: only; I’ll add a case-insensitive mailto: exemption and regression test.

The audit currently exempts javascript: only; I’ll add a case-insensitive mailto: exemption and regression test.

run$ apply_patch <<'PATCH' *** Begin Patch *** Update File: lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js @@ }) .filte…
apply_patch <<'PATCH'
*** Begin Patch
*** Update File: lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
@@
       })
       .filter(anchor => {
-        // Ignore href's that are not real links
-        return !anchor.href || !anchor.href.toLowerCase().startsWith('javascript:');
+        // Ignore href's that are not real links or do not navigate to a page.
+        const href = (anchor.href || '').toLowerCase();
+        return !anchor.href || (!href.startsWith('javascript:') && !href.startsWith('mailto:'));
       })
*** Update File: lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
@@
   it('does not fail for links with javascript in href attribute', () => {
     const auditResult = ExternalAnchorsAudit.audit({
       AnchorsWithNoRelNoopener: [
         {href: 'javascript:void(0)'},
@@
     assert.equal(auditResult.rawValue, true);
     assert.equal(auditResult.details.items.length, 0);
   });
+
+  it('does not fail for links with mailto in href attribute', () => {
+    const auditResult = ExternalAnchorsAudit.audit({
+      AnchorsWithNoRelNoopener: [
+        {href: 'mailto:sales@example.com'},
+        {href: 'MAILTO:support@example.com'},
+      ],
+      URL: {finalUrl: URL},
+    });
+    assert.equal(auditResult.rawValue, true);
+    assert.equal(auditResult.details.items.length, 0);
+  });
 });
*** End Patch
PATCH
output · 9 lines
Wall time: 0.0000 seconds
Output:
Exit code: 0
Wall time: 0 seconds
Output:
Success. Updated the following files:
M lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
M lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
messagePatch is in; I’ll run the focused audit test and inspect the final diff.

Patch is in; I’ll run the focused audit test and inspect the final diff.

run$ ./node_modules/.bin/mocha lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js --timeout 10000
./node_modules/.bin/mocha lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js --timeout 10000
output · 18 lines
Chunk ID: 648f78
Wall time: 0.2552 seconds
Process exited with code 0
Original token count: 92
Output:
​
​
  External anchors use rel="noopener"
    ✓ passes when links are from same hosts as the page host
    ✓ fails when links are from different hosts than the page host
    ✓ handles links with no href attribute
    ✓ does not fail for links with javascript in href attribute
    ✓ does not fail for links with mailto in href attribute
​
​
  5 passing (20ms)
run$ git diff --check && git diff -- lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js lighthouse-core/test/audits/dobette…
git diff --check && git diff -- lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js && git status --short
output · 45 lines
Chunk ID: 59373a
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 510
Output:
diff --git a/lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js b/lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
index 296f94d..21b1147 100644
--- a/lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
+++ b/lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
@@ -47,8 +47,9 @@ class ExternalAnchorsUseRelNoopenerAudit extends Audit {
         }
       })
       .filter(anchor => {
-        // Ignore href's that are not real links
-        return !anchor.href || !anchor.href.toLowerCase().startsWith('javascript:');
+        // Ignore href's that are not real links or do not navigate to a page.
+        const href = (anchor.href || '').toLowerCase();
+        return !anchor.href || (!href.startsWith('javascript:') && !href.startsWith('mailto:'));
       })
       .map(anchor => {
         return {
diff --git a/lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js b/lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
index c89aab4..ea8d777 100644
--- a/lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
+++ b/lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
@@ -66,4 +66,16 @@ describe('External anchors use rel="noopener"', () => {
     assert.equal(auditResult.rawValue, true);
     assert.equal(auditResult.details.items.length, 0);
   });
+
+  it('does not fail for links with mailto in href attribute', () => {
+    const auditResult = ExternalAnchorsAudit.audit({
+      AnchorsWithNoRelNoopener: [
+        {href: 'mailto:sales@example.com'},
+        {href: 'MAILTO:support@example.com'},
+      ],
+      URL: {finalUrl: URL},
+    });
+    assert.equal(auditResult.rawValue, true);
+    assert.equal(auditResult.details.items.length, 0);
+  });
 });
 M lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
 M lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
thinkingChecking unit test results

The unit test shows it can handle both lower and uppercase, and I have 5 validation tests passing, which is great! I don't see the need for any more tests at this point. However, I'm a bit concerned that the status output indicates only two modifications, but I'll keep an eye on that. Overall, it seems I can finalize with up to 10 lines.

messageUpdated the rel="noopener" audit to ignore mailto: links, case-insensitively, in lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:50. Ad…

Updated the rel="noopener" audit to ignore mailto: links, case-insensitively, in lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:50. Added lowercase and uppercase regression coverage in lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js:69.

Validation: all 5 focused audit tests pass.

The screenshots (a Lighthouse audit flagging the mailto link, and a broken link on Android) never come up. The title alone says what to fix, and the existing javascript: exemption in the code says how.

Results

Roughly the same results were obtained in the three cases. Around 175-178 pass scores out of 480 which turns out to be roughly 37%. This is in close territory to the 37.5% that vals.ai's SWE-bench Multimodal leaderboard reports for GPT 6 Luna on max effort (as of October 2026), even though they run it in a different harness (mini-swe-agent).

Run Resolved Rate
Native images 177 / 480 36.9%
OCR text 175 / 480 36.5%
No images 178 / 480 37.1%

Similar totals could still come from solving different tasks, so here is the per-task comparison (tasks that weren't graded count as failures, same as in the totals):

Pair Both resolved Only the first Only the second Both failed
Images vs OCR 163 14 12 291
Images vs no images 167 10 11 292
OCR vs no images 161 14 17 288

157 tasks were resolved in all three runs and 284 failed in all three, so 441 of 480 (92%) got the same result every time. The other 39 flip in both directions in about equal numbers (McNemar p ≥ 0.7 for every pair).

Auditing the traces

To further check if the model was looking up images via some other means that I was missing, I sacrificed my Claude Max sub and put on several Sonnet 5.5 reviewers to go through the traces line by line. The Sonnet 5.5 reviewers pointed out

  1. There were no network lookups.
  2. In the OCR and no image runs, there were mentions of OCR and placeholders respectively in the chain of thought.
  3. Sonnet 5.5 reviewers said that "Images were almost not required" to solve the tasks. I am directly pasting it's legible output below. These are the reviewers' ratings, not a property of the dataset; they read each native-image trace with the issue images, the merged fix and the hidden tests in front of them.

Images were almost not required

In 403 of 480 tasks (84%) the issue text fully specifies the bug and the images only illustrate it. Resolution rates are flat across the three runs in every category, so the images don't change outcomes even where they add information.

Reviewer rating Tasks Resolved: images / OCR / omitted
not_needed: text fully specifies the task 403 167 / 166 / 169
helpful: text is enough, images remove ambiguity 70 9 / 9 / 9
required: needed information is only in the images 7 1 / 0 / 0

No task needed an image file from the repository itself; only the issue screenshots mattered at all.

The seven required tasks, of which only one is an image win (click to expand)
TaskImages / OCR / omittedWhat only the images showWhy the native run still failed
carbon-12384R / F / FSpec: read-only steppers stay visible but greyedResolved. The only clear image win in 480 tasks
lighthouse-15092F / F / FWhole error redesign; issue text is the title onlyTests check the category gauge, which Luna left alone
lighthouse-9151F / F / FWhich gauge is broken (PWA badge)Patched the path shape; the cause was duplicate SVG ids
lighthouse-4301F / F / FGatherer names during the hang point to start-urlPatched driver listener setup instead of the gatherer
carbon-3139F / F / FThe two accessibility violationsRight fix, but undefined where snapshots expect null
carbon-4260F / F / FThe two accessibility violationsSnapshot expects an id scheme nobody specified
openlayers-14100F / F / FWMTS GetTile URLs with TIMERead it as a throttling question; no patch

The hidden tests don't check what the images show

Another thing that came up in the audit is that the hidden tests mostly don't check what the images show. A lot of the Carbon issues come with detailed design spec screenshots (colours, spacing, icon placement), but the test that grades them is often just a snapshot of the component's prop types (carbon-13851, for example, comes with a full TimePicker error/warning spec, and the test only checks that two new props exist). The OpenLayers and bpmn-js issues have GIFs of the bug, while the tests check internal state like waypoints or function arguments.

When tasks failed, it was usually because the tests expected some detail that neither the text nor the images mention, like an exact prop name, null vs undefined, or something settled in the GitHub issue comments that the model never sees. The Sonnet 5.5 reviewers flagged this for almost every failed task, though I have not manually verified this.

For the images to matter, a test would have to check something you can only learn from them, like which component is broken or a value from the design spec. That is happening rarely here.

Ten tasks you can check yourself

Here are ten tasks where you can judge for yourself. Each was resolved in all three full runs: with the images, with OCR text instead, and with no images, and the no-image transcript shows no image reaching the model. Two Fable 5.1 reviews per task (both saw the issue and its images; one also got the merged fix and hidden tests, the other the agent's transcript with the result removed) rated the images "not needed" in all 20 cases. Since I picked these from tasks that passed everywhere, they show that solving from text alone happens, not how often; the 403/480 above is the estimate for that.

Task What the images show What the text already says
lighthouse-4036 The noopener audit flagging mailto links; Chrome for Android opening a blank page Title: the audit "should ignore href="mailto:x@y.com" links". The repo already skips javascript: links the same way
prism-1602 Rendered YAML where "world" loses its colour when followed by # test "String highlighting does not work when a string has a comment on the same line", plus the repro - "world" # test
next-2984 Console TypeError "Cannot add property value, object is not extensible" CascaderSelect changes the props passed in; "the component should not change the user's values"
bpmn-js-1655 Diagram whose external labels are drawn in the pale fill colour "External labels should be set to the 'stroke' color, as seen in the embedded labels"
carbon-6675 Pagination footer reading "1 of 1 pages" Names the file, the lines and the rule: singular when max === 1 or totalPages === 1
eslint-14242 Terminal: exit code 1 without --quiet, exit code 0 with it "The exit code should be 1 when max-warnings is exceeded"
grommet-6282 DataTable sorted by Parent, rows with null Parent not grouped "Undefined properties should be treated as empty string"
highlight.js-2958 Output where <a:FAIL …> is plain text and the other tags are coloured Sample code labels each line OK or FAIL; title: "single letter namespace prefix not highlighted"
openlayers-14627 WebGL stack trace; debugger in fromEPSG4326 with Infinity inputs Repro code, the cause ("Math.tan will transform Infinity to NaN") and the expected behaviour
prettier-11884 Editor showing a lone comma inside [ ] Input, CLI options, the bad output and the expected output

You can read the full no-image traces of all ten tasks here: every reasoning summary, message and command with its output, plus the issue images the model never saw.

A run on MMMU-Pro as a control

We (me and Florian) were still not 100% convinced that this was happening so I ran the same setup of native image input, customised OCR and placeholder image run on MMMU-Pro, with the same model, reasoning effort and interceptor.

MMMU-Pro is a good control: it is not agentic, just single-turn multiple-choice questions across college subjects, with no repo or tools to fall back on. In it's standard split, each question comes with a diagram, chart or photo that carries information the text does not restate, so a model that cannot see the image should actually lose points there.

We did get different results this time.

Run Accuracy Correct Output tokens / question
Native images 79.7% 1378 / 1730 1.7k
OCR text 56.3% 974 / 1730 4.1k
No images (placeholder) 36.9% 638 / 1730 2.1k

Every gap is large and paired (McNemar p < 1e-49 for all three pairs). Taking the images away costs 43 points here; on SWE-bench Multimodal it cost nothing. The OCR run also burns more than twice the output tokens of the native run (reasoning tokens alone: 4.0k vs 1.6k per question), which I interpret as the model trying to understand/reconstruct the figure from text.

The drop is biggest where the image carries structure that words don't: Art goes from 42/53 to 3/53 without images, Finance from 59/60 to 15/60, Accounting from 51/58 to 10/58. It is smallest where the text already carries the question: Literature 43/52 → 38/52, Energy and Power 50/58 → 38/58.

MMMU-Pro results by subject (click to expand)

Sorted by how much accuracy drops when the images are removed. Cell colour goes from red (low) to green (high). The biggest drops are highlighted.

SubjectNativeOCRNo imagesDrop (native → none)
Art79%
42/53
34%
18/53
6%
3/53
−74 pts
Finance98%
59/60
88%
53/60
25%
15/60
−73 pts
Accounting88%
51/58
83%
48/58
17%
10/58
−71 pts
Design83%
50/60
40%
24/60
20%
12/60
−63 pts
Marketing95%
56/59
71%
42/59
34%
20/59
−61 pts
Computer Science77%
46/60
42%
25/60
20%
12/60
−57 pts
Public Health86%
50/58
81%
47/58
31%
18/58
−55 pts
Art Theory84%
46/55
49%
27/55
31%
17/55
−53 pts
Biology78%
46/59
42%
25/59
27%
16/59
−51 pts
Chemistry82%
49/60
47%
28/60
32%
19/60
−50 pts
Geography79%
41/52
48%
25/52
29%
15/52
−50 pts
Management72%
36/50
48%
24/50
22%
11/50
−50 pts
Materials82%
49/60
53%
32/60
32%
19/60
−50 pts
Economics90%
53/59
64%
38/59
41%
24/59
−49 pts
Physics85%
51/60
52%
31/60
42%
25/60
−43 pts
Architecture and Engineering83%
50/60
62%
37/60
43%
26/60
−40 pts
Electronics97%
58/60
63%
38/60
57%
34/60
−40 pts
Music63%
38/60
30%
18/60
25%
15/60
−38 pts
Basic Medical Science83%
43/52
40%
21/52
46%
24/52
−37 pts
Math87%
52/60
63%
38/60
53%
32/60
−33 pts
Psychology72%
43/60
48%
29/60
38%
23/60
−33 pts
Pharmacy79%
45/57
63%
36/57
47%
27/57
−32 pts
History70%
39/56
68%
38/56
39%
22/56
−30 pts
Clinical Medicine73%
43/59
46%
27/59
44%
26/59
−29 pts
Sociology74%
40/54
61%
33/54
46%
25/54
−28 pts
Agriculture57%
34/60
45%
27/60
33%
20/60
−23 pts
Mechanical Engineering71%
42/59
61%
36/59
49%
29/59
−22 pts
Energy and Power86%
50/58
71%
41/58
66%
38/58
−21 pts
Diagnostics and Laboratory Medicine55%
33/60
47%
28/60
38%
23/60
−17 pts
Literature83%
43/52
77%
40/52
73%
38/52
−10 pts

Conclusion

We have a couple of strong pointers indicating that, for almost all SWE-bench Multimodal tasks, the images don't carry information the model needs:

  1. The results of the three runs (native images, OCR and no images) are roughly the same, 175–178 out of 480, and 92% of tasks get the same result in all three.
  2. The Sonnet 5.5 reviewers (and Fable 5.1 reviews for a few tasks) say that the images are not really needed to do the tasks, apart from a handful (the seven required ones above). I trust my clankers.
  3. The same setup on MMMU-Pro drops 43 points when the images are removed. So the setup does take the image information away, and Luna does use images when they matter. On SWE-bench Multimodal, removing them barely changes anything.

Possible caveats

I have run only GPT 6 Luna xhigh based on budget and rate limit constraints on the benchmark. Ideally, one should also run Claude models and potentially DeepSeek Flash with image support off.

I say this because if you look at the leaderboard, Claude models are well ahead of OpenAI models on this bench (Opus 5.5 at 70.2%, Fable 5.1 at 61.3%, Sonnet 5.5 at 58.3%, against 43.5% for the best OpenAI model). MMMU-Pro shows that Luna can use images, but it could still be that OpenAI models don't utilize the images much while working as an agent in a repo. It would be great if someone with the budget, e.g. the Vals AI people, could check this. This is currently out of my time and cost budget. Opus 5.5 runs are very expensive lol (and I don't want to spend my PI inference credits on it).

Another possibility is memorisation. The tasks come from public GitHub issues and PRs from 2017-2024, so Luna may already know some of the fixes, which would also make the images unnecessary. The MMMU-Pro control doesn't rule this out.

If you replicate this, with Luna or any other model, please let me know what you find, whether you find that the images are being utilized or not.