Agents Can't Iterate Against Tests That Lie

Rocky Warren, Senior Staff Software Engineer

April 21, 2026 16 min read

Between February 2025 and February 2026, coding agents went from writing none of Clipboard's code to nearly all of it.

Seemingly overnight, agents made writing code cheap. More code amplified our flakiness. A flake is a test that exhibits both a passing and failing status across multiple runs for the same commit.

At one point, 100% of PRs in two of our largest repositories experienced at least one flaky test. It's difficult to use your continuous integration (CI) pipeline as a verification gate when it randomly swings open and shut, regardless of code quality.

We drove our end-to-end (E2E) flake rate from 100% to under 15% in six weeks; still not ideal, but 85% less absurd. We achieved this through agent-driven test analysis, migrating to more reliable tests, and building custom error reporters optimized for agent readability.

Agents shifted our focus from writing code to verifying it. When humans write code, flaky tests are annoying. When agents write code, flaky tests break the feedback loop that keeps agents moving at full speed.

The vicious cycle

On each pull request (PR), our CI pipelines run checks and unit, integration, and E2E tests.

E2E tests are typically the flakiest because an issue in any of our 30+ backend services, or a one-off network issue, a slow database query, or an eventual consistency blip, can cause test failures.

Flaky tests are the kind of problem that's easy to ignore when humans write code. You configure auto-retries or rerun the job, mutter something about timing, and move on with your day. When agents write your code and the process relies on automated verification, flakiness blocks progress. Agents can't iterate against a test suite that lies to them.

As agents entered the fray, we got trapped in a vicious cycle: More code means more PRs, which means more load on and more data in our E2E staging environment. The increased load and data sizes can cause flakiness, leading to retries, even more load and data, and yet more flakiness.

Because there are so many failure modes, it's difficult to give agents the context they need to fix the flakes. Provide an agent the error message, stack trace, and frontend repo, and nearly all will add a retry or increase timeouts. This may solve the immediate issue, but we need long-term solutions.

With these challenges defined, we split our focus between two questions: 1) How many E2E tests do we actually need? Assuming the answer is anything greater than zero, we'll have some amount of flakiness, so 2) How do we get agents enough context to root cause E2E failures?

1. How many E2E tests do we actually need?

We had 174 E2E tests. Is this too many, not enough? To answer that question, we first needed to answer, "Why do we write tests?" We write them to protect our customers from bugs and regressions that, in Clipboard's case, impact patient and student care and worker income.

Where sufficiently possible, we should validate using more reliable test types. The E2E tests we write should justify their higher maintenance burden by validating what only they can: The integration of our system as a whole, especially along critical paths.

Multi-agent consensus for test triage

To answer, "Is 174 E2E tests the right amount?", we:

  1. Gave three agents using separate models and harnesses a detailed prompt to investigate and categorize each E2E test with a reason and confidence score. The categories:
    1. delete because they are fully covered by other tests.
    2. combine with other tests, referencing the test(s) to combine into.
    3. convert to integration or unit tests.
    4. leave as legitimate E2E tests.
  2. Combined the resulting files into a single file, randomizing and obfuscating which agent came to which conclusion.
  3. In fresh context windows, had two agents using separate models and harnesses make a final decision for each test based on the original research, inviting them to explore the code and validate claims as necessary.
  4. Had the two agents check each other's work and go back and forth until they reached a consensus for each test.

The final JSON file showed we may be able to maintain our existing level of coverage while reducing our 174 E2E tests to 46, a 74% reduction.

Example output:

{
  "playwright/e2e/auth/onboarding.spec.ts": {
    "should start the onboarding flow for new phone numbers and valid one-time passwords": {
      "confidence": "high",
      "decision": "delete",
      "reason": "Other tests perform identical phone signup → OTP → redirect flow; this is a strict subset."
    },
    "should log in via phone number with a valid one-time password": {
      "confidence": "high",
      "decision": "leave",
      "reason": "Canonical phone login test: user creation → OTP → authenticated page navigation."
    }
  }
}

Is dropping so many E2E tests while agents simultaneously write code faster than ever a good idea? That depends on how much signal our tests provide. In practice, because of their flakiness, we were quick to assume downstream issues and repeatedly hit the "Re-run failed checks" GitHub button. With 100% of PRs experiencing at least one flaky test, we'd lost confidence.

Not everyone agreed. When we proposed the reduction, an argument was, "Flakiness is an industry-wide problem, but not often to the extent that the best solution is to give up and delete tests. Shouldn't we address the underlying flakiness instead?"

We detailed the previous and ongoing efforts to address flakiness. Revisiting the test suite wasn't giving up. Agents proposed deleting 38 tests that were fully covered elsewhere, combining 6 tests with overlap, converting 82 tests to integration tests, and keeping the remainder. With this breakdown, stakeholders agreed.

For implementation, we:

  1. Split the final JSON output by test file to limit merge conflicts.
  2. Gave each set to coding agents to implement and open PRs.
  3. Had the agents determine the owning team and assign them as code reviewers.

In the end, we ended up with 87 E2E tests, a 50% reduction, but higher than the 46 agents originally proposed. Domain owners pushed back on specific cuts. One engineer argued that while individual E2E tests rarely catch regressions, they prevent breaking changes to backend APIs that cause outages. "If we cut them, we lose this safety net and start to rely only on the reviewer to catch these cases. With the flood of AI code, this is already harder than before."

They were right. We needed coverage for the frontend-backend interface, and kept many of the tests. There are more reliable ways to prevent API-breaking changes, and the ensuing discussion helped prioritize a project that is now underway.

We immediately saw a reduction in E2E flakes per PR, dropping from 100% to under 15% in six weeks.

To prevent regressions, we updated AGENTS.md files, providing agents a checklist before creating E2E tests and encouraging, "when in doubt, write an integration test instead."

2. How do we get agents enough context to root cause E2E failures?

Deleting, converting, and combining E2E tests gave us the largest reduction in flakiness. But we still had 87 E2E tests, and they still fail.

The latest agents are great at root-causing our unit and integration tests, where they have full context and tight feedback loops: they can run the test 50 times in a loop to verify their fix.

Our E2E tests run against a deployed environment with background jobs, async messages, and eventual consistency across 30+ backend services. When agents see a "Timeout 30000ms exceeded" error message and can't find an obvious bug, they can only offer to add retries or longer timeouts.

We use Playwright for our E2E tests. When one fails, and a human is debugging it, they'll usually download a Playwright report. Depending on the configuration, it might include video, screenshots, console logs, and network traces. From there, they have context into the backend services and can use tools like Datadog's Application Performance Monitoring (APM) to trace requests and find issues.

Our first attempt was to write a skill that helped agents download the HTML report and inspect it. While it improved things, the report isn't built for machines to easily inspect.

We looked into writing tools to pull data from the Playwright Blob and HTML reports, but any upstream changes to those formats could break our parsing. We needed a custom reporter to control the format.

We built @clipboard-health/playwright-reporter-llm, which outputs structured JSON for agent readability. It outputs error messages, a unified timeline of steps, network, and console events, and base64-encoded failure screenshots.

The reporter was able to confidently root cause frontend-only failures. To get a more complete picture, we needed access to APM traces. From each backend service, we emit the W3C Trace Context traceparent response header, a vendor-neutral format that works with Datadog, OpenTelemetry, etc. It shows up in our Playwright reporter at tests[].attempts[].network[].responseHeaders, and the reporter parses it into first-class traceId and spanId fields.

app.use((_request, response, next) => {
  const span = tracer.scope().active();
  if (span) {
    response.setHeader("traceparent", span.context().toTraceparent());
  }
  next();
});

Putting the pieces together, we created a /flaky-test-debugger skill that tells agents how to download the custom Playwright report, what it contains, how to parse it, and how to look up trace IDs in Datadog to provide complete context for what happened during the test.

Don't underestimate agent selection. We gave the same test failures and prompts to three separate agents: Devin.ai, Claude Code (Opus 4.7 xhigh), and Codex (gpt-5.4 xhigh). While we primarily use Claude Code for day-to-day coding, Codex consistently performs best for this task. In our experience, it searches deeper for context, runs queries more quickly to validate its assumptions, and, as a result, is least likely to resort to retries and timeouts as a "fix".

Examples

I gave the agents two prompts similar to the following:

Invoke .agents/skills/flaky-test-debugger/SKILL.md.

{
  "test": "Shift block cancellation should cancel booked block",
  "file": "e2e/shiftBlocks/cancelBlocks.spec.ts:14",
  "failures": [{
    "error": "Timeout: locator.click exceeded 3000ms waiting for getByRole('button', {name: /Yes, cancel/})",
    "pipeline": "https://github.com/[...]/actions/runs/23929364/attempts/1",
    "commit": "2208032d652be20c89dc06152d78930ce15269a0"
  }]
}

One agent's solutions:

- await page.getByRole("button", { name: /Yes, cancel/i }).click();
+ await expect(async () => {
+   await page.getByRole("button", { name: /Yes, cancel/i }).click({ timeout: 5_000 });
+ }).toPass({ intervals: [1_000, 2_000], timeout: 15_000 });

+ const newPagePromise = page.context().waitForEvent("page");

- const [newPage] = await Promise.all([
-   page.context().waitForEvent("page"),
-   page.waitForLoadState(),
- ]);
+ const newPage = await newPagePromise;

In contrast, Codex provided trace evidence in each case pointing to actual product bugs and recommended fixes spanning 5 and 9 files, respectively. The third agent was somewhere in the middle.

Codex's PR summary for one of the fixes:

The flaky cancellation test was timing out while waiting for an actionable
"Yes, cancel" button after the dialog copy had already rendered. The old
structure kept `useModalState` inside the cancel trigger subtree, so a remount
of that subtree could cause the open dialog state to be dropped. Lifting the
modal state into `ShiftBlockDetails` removes that reset path, and the E2E now
synchronizes with the actual footer buttons, becoming enabled before clicking.

What we learned

We'll continue to iterate and increasingly automate to root cause and fix flaky tests as early in their lifecycle as possible.

This process forced us to be honest about what we're expecting from our tests, and that "delete" is a valid answer. Code is a liability, and tests usually get a pass because "coverage is good." Each test has a maintenance cost, and you pay the highest cost for lying tests.

Use our @clipboard-health/playwright-reporter-llm and /flaky-test-debugger skill if you're hitting the same wall, and let us know what you think!