Let an AI Agent Write Your Selenium Tests in Java (and Where It Goes Wrong)
A measured look at agent-style first-draft Selenium tests in Java: 120 headless-Chromium runs show the draft passing 20 of 30 on the page it was written for and 0 of 30 after a redesign, and a repaired suite passing 60 of 60. Honest about what was and was not an agent.
Ask an AI coding agent to “write Selenium tests for this page” and it will hand you something that looks right within seconds: it finds buttons, clicks them, asserts on text. It is a reasonable first draft. The interesting question is not whether the tests compile; it is how long they stay green, and whether the failures you meet are the kind a person would have written too.
This post measures that on a small shop page with six tests. The same six tests run 5 times each against the page they were written for, and again after a plausible redesign of that page, then again after a repair pass. Everything is plain Selenium 4 and JUnit in Java; the full source is linked beside each block.
Read this first: what was and was not an agent. The “first draft” tests in AgentDraftTest.java were written by me (Claude) in this session, from the v1 page source, the way an agent would write them, and then run unedited. I did not drive Copilot, Cursor, Claude Code as a separate tool, or any other product, so the pass rates below are not a score for any of them. They show the class of failure a first draft like this produces. The repair pass was also mine, and I knew the answers because I designed the redesign; a real agent would have to discover them from failure output. And I did not run Playwright, so nothing here is a measurement about it.
Versions. JDK 25 (Temurin 25.0.4.1+1), Selenium 4.51.0, JUnit 6.1.3, headless Chromium 141 with a matching chromedriver 141, Surefire 3.5.4. The sandbox had a Chromium 141 binary and a chromedriver 147; they do not match, so I downloaded chromedriver 141 from Chrome for Testing. All 120 results: selenium-agent-results.csv; the summary: selenium-agent-summary.txt. There is no companion repository for this post; the four Java files are in the media library.
What goes wrong first: the draft is already flaky on the page it was written for
Before any redesign, on the very page the tests were written against (v1), the draft is not reliable. Six tests, five runs each, one headless Chromium per run:
AgentDraftTest on v1: 20/30 runs passed
AgentDraftTest on v2: 0/30 runs passed
RepairedTest on v1: 30/30 runs passed
RepairedTest on v2: 30/30 runs passed
Those four lines are the first block of selenium-agent-summary.txt, written by the run’s own summariser. The shape of the whole post is in them: the draft passes two thirds of the time on v1, never on v2, and the repaired suite passes every time on both. Per test on v1:
Three tests are solid, and three are flaky or broken. The flaky ones have one thing in common: they click a button and then check the cart badge. Here is why.
The empty bar on the right of the draft is the argument: a redesign that changes no behaviour at all — a user can still search, add to cart and see prices — took the draft from two thirds to nothing.
The mental model: tests that describe markup versus tests that describe behaviour
A Selenium test finds elements on the page by a locator (“the element with id q“, “the second button with text ‘Add to cart'”) and then acts on them. A locator that describes the markup (class names, ids, position, exact button text) breaks when the markup changes. A locator that describes a stable role the page promises to keep (a data-testid attribute added for tests) survives redesigns. A first draft written from page source naturally reaches for whatever is visible in the source, which is markup.
The second half of the model is time. After a click that triggers a network call, the page changes later. A test either waits for the condition it needs, or guesses how long to sleep.
To make the comparison fair, a tiny server (the JDK’s built-in HTTP server, in ShopServer.java) serves the same shop twice. /v1 is the page the tests are written against. /v2 is the same shop after a redesign: a promo banner above the header, renamed classes, the button label changed from “Add to cart” to “Add”, no element ids, no Search button (the list filters as you type), and prices shown as ₹120.00 instead of ₹120. The add-to-cart request takes between 80 and 419 ms, like a real backend, so the cart badge updates after the click returns. Every element carries a data-testid in both versions; that is the contract the repaired tests use, and it is an assumption I made, not something your application necessarily has.
From ShopServer.java. The classes, tag names, label and price format all differ; the data-testid values are identical. BrowserSupport.java starts headless Chromium for each test and appends one PASS/FAIL line per run to the CSV.
Failure 1: sleeping instead of waiting
The draft’s click-then-check pattern:
@RepeatedTest(5)
void addTwoProductsShowsTwoInCart() throws Exception {
driver.findElement(By.xpath("(//button[text()='Add to cart'])[1]")).click();
driver.findElement(By.xpath("(//button[text()='Add to cart'])[2]")).click();
Thread.sleep(300);
assertEquals("2", driver.findElement(By.id("cart-count")).getText());
}
From AgentDraftTest.java. Thread.sleep(300) is a guess: “the server usually answers within 300 ms”. With a backend that takes 80 to 419 ms per call and two calls in flight, the guess is right sometimes. Result across five runs: it passed once. The failures, from the run summary:
v1 AgentDraftTest addFirstProductShowsOneInCart x1: AssertionFailedError: expected: <1> but was: <0>
v1 AgentDraftTest addTwoProductsShowsTwoInCart x4: AssertionFailedError: expected: <2> but was: <1>
v1 AgentDraftTest addingThePenThreeTimesShowsThree x3: AssertionFailedError: expected: <3> but was: <1>
v1 AgentDraftTest addingThePenThreeTimesShowsThree x2: AssertionFailedError: expected: <3> but was: <2>
From selenium-agent-summary.txt. The failure messages say what happened without saying why: the badge showed 1 when the test wanted 2, because the second request had not finished yet. The tell-tale is that the “actual” value is lower than expected and the same test passes on another run. The test that clicks the same button three times failed all five runs, since three sequential 80–419 ms calls rarely finish inside 300 ms.
Do not “fix” this by raising the sleep. A longer sleep makes the test slower for everyone and still fails the day the backend is slow. The fix is to wait for the condition (below). If an agent proposes Thread.sleep(1000) when you show it this failure, that is the wrong repair.
Failure 2: locators that describe markup
On v2 the draft fails every time. The failure messages (the long ones are cut off by the summariser, exactly as in the file):
v2 AgentDraftTest addFirstProductShowsOneInCart x5: NoSuchElementException: no such element: Unable to locate element: {"method":"css selector";"selector":".card
v2 AgentDraftTest addTwoProductsShowsTwoInCart x5: NoSuchElementException: no such element: Unable to locate element: {"method":"xpath";"selector":"(//button[tex
v2 AgentDraftTest searchingForPenLeavesOneProduct x5: NoSuchElementException: no such element: Unable to locate element: {"method":"css selector";"selector":"#q"}
v2 AgentDraftTest emptySearchShowsAllThreeProducts x5: AssertionFailedError: expected: <3> but was: <0>
.card no longer exists (it is .product-tile), the button text is “Add” not “Add to cart”, and the search input has a name but no id. Three of these are loud and easy to diagnose: NoSuchElementException names the locator. The fourth is quieter: emptySearchShowsAllThreeProducts finds nothing by class name and reports “expected 3 but was 0”, which reads like a behaviour bug (the page shows no products) when it is a locator that matches nothing. Always check the locator before believing a count of zero.
Failure 3: asserting on formatted text
The most instructive failure is the one where the element is found:
v2 AgentDraftTest notebookCosts120Rupees x5: AssertionFailedError: expected: <₹120> but was: <₹120.00>
The draft compared the price as a string: assertEquals("₹120", price). After the redesign the page shows ₹120.00. The price did not change; its formatting did. A locator-repair loop will not fix this, because the locator already works, and a naive “make the test pass” repair might change the expected string to ₹120.00, hard-coding the new formatting. The better assertion is on the value: extract the number and compare it numerically, which is what the repaired test does.
From RepairedTest.java. Locate by data-testid; WebDriverWait with textToBe polls until the badge shows the expected count (up to 5 seconds) instead of sleeping a fixed time; the price is parsed to a BigDecimal so 120 and 120.00 are equal. Result: 30 of 30 on v1 and 30 of 30 on v2. The explicit wait also lets a test continue the moment its condition is true, though I did not time that. The one file passes on both page versions, which is the point of the exercise.
The caveat that makes this repair look easier than it is. It only works because both pages already had data-testid attributes. If your application has none, the repair requires a change in the application, not just in the tests, and that is a conversation with the front-end team. An agent can propose the attributes, but someone has to add them and promise to keep them.
Using an agent well for this
What the measurements suggest, stated as advice rather than as an experiment I ran:
Give the agent the contract, not just the HTML. Tell it to use data-testid locators and explicit waits. A first draft written from page source alone chose ids, classes, and XPath on text, as mine did.
Make it run the tests several times. The v1 draft would look fine after one run (the “passes once” test passes 1 in 5; a lucky run tells you nothing). The @RepeatedTest(5) in these files is what exposed the flakiness.
Read the failure output before accepting a repair. “Expected 3 but was 0” and “expected ₹120 but was ₹120.00” both invite the wrong fix.
Review the diff of the repair. A repair that lengthens a sleep, loosens an assertion, or deletes a test has made the suite green, not correct.
For more on setting an agent up so it does not take shortcuts like these, see the working-agreement post: AI coding agents on a Spring Boot codebase.
Reference: how the 120 runs were produced
Each cell of the matrix is one Maven invocation with system properties that the pom forwards to Surefire: -Dui.version=v1|v2 picks the page, -Dresults.file=... is the CSV the ResultLog extension appends to, -Dchrome.binary=... and -Dchromedriver.path=... point at the matching browser and driver. In each cell JUnit runs every test five times (@RepeatedTest(5)) and a fresh headless Chromium is started per run. The pom:
A trap that cost a cycle: JUnit’s @ExtendWith needs an extension class with a public no-argument constructor, so the result logger cannot take a suite name as a constructor parameter; it derives it from ctx.getRequiredTestClass() instead. Another: a chromedriver newer than the browser (147 versus 141 here) refuses to start the session; match them, or let Selenium Manager fetch a pair if your network allows it (mine did not).
What about Playwright?
The brief for this post mentioned Playwright, and I want to be straight that I did not run it. What can be said without a measurement: the Playwright for Java locators page describes locators as “the central piece of Playwright’s auto-waiting and retry-ability”, which targets exactly the sleep-versus-wait failure in this post, and says “Testing by test ids is the most resilient way of testing” while noting that test ids are not user-facing and that role and text locators are the alternative when that matters. Whether that makes agent-written Playwright tests more robust than agent-written Selenium tests is an experiment I have not done. See the Playwright locators documentation.
Should you even do this? Yes for the first draft: it is quick, and it finds the flows worth testing. No for trusting the draft: expect flaky waits, markup-coupled locators and string-formatted assertions, and budget for a human (or a second, better-instructed pass) to harden them. Keep the number of browser tests small, since each one takes seconds and every one is a maintenance cost; push logic checks down to unit and API tests. And remember the limits of this experiment: one small page, one browser, a first draft written by me and not by a product, and a repair made with hindsight.
No Comments yet!