End-to-End Testing
A real browser over the flows you cannot ship broken — signup, login, checkout, payment, upload, critical navigation — and an honest account of what that costs.
The intent, the obvious build, and why it breaks
Every lesson starts where the work starts: someone wanted an outcome, and the first implementation that comes to mind has a problem.
Which flows justify a real browser driving a real stack, and what will that suite cost me every week for the rest of the project?
A person wants to sign up, log in, put something in a basket, pay, and get what they paid for. If any of those is broken, nothing else about the product matters.
Automate the whole product in a real browser. It is the closest thing to a real user, so the more of it we cover, the more confident we can be — and when a test is flaky, retry it a couple of times.
Coverage in a browser is bought at browser prices. A suite that covers everything takes tens of minutes, runs after the fact rather than before the merge, and stops being part of anyone's loop.
- Coverage in a browser is bought at browser prices. A suite that covers everything takes tens of minutes, runs after the fact rather than before the merge, and stops being part of anyone's loop.
- Every test in the suite depends on the whole stack, so one slow dependency or one migration makes all of them fail at once, and the failure says nothing about which change caused it.
- Flake is not a nuisance here, it is the dominant cost. A test that fails one run in twenty is a coin flip on a suite of a hundred tests, and the team learns to press re-run before reading the failure.
- Retries are the trap. A test that passes on the second attempt has usually just demonstrated a real race — the same race a user will lose one time in twenty — and the retry deletes that evidence (Out-of-Order Responses).
- The suite accumulates state: accounts that already have orders, feature flags left flipped, a seeded database that drifts from the schema. It fails for reasons that have nothing to do with the product.
What is actually happening
In the browser, not in the framework.
- The runner drives a real browser through a protocol, performing real navigations, running the real bundle, and letting the real network layer talk to a real server (Client-Side Routing).
- What this buys is the integration: routing, auth, cookies and storage, hydration, service workers, request handling and everything that only exists once the pieces are joined (The Service Worker Lifecycle).
- What it costs is that every asynchronous boundary in the system is now inside the test. The test does not know when the application is ready; it can only observe the DOM and infer.
- Modern runners auto-wait on queries rather than sleeping, which removes the largest single source of flake. The remaining flake is state: what the previous test left behind, what another worker is doing to the same account, what the backend cached (Interleavings: The Schedule Is Part of the Program in Concurrency).
- Test isolation is the load-bearing property. Every test needs its own account, its own data and its own storage, or the suite becomes an ordering puzzle (localStorage and sessionStorage).
- A flaky end-to-end test is usually a real race that a person hits rarely: a click landing during a re-render, a form submitted before a token refresh completed, a navigation racing a redirect (Reasoning About Races: A Method, Not an Instinct in Concurrency).
What this makes the browser do
And which of it is avoidable.
- A full browser process per worker, with the entire pipeline running for every page of every test — parse, style, layout, paint, composite, over and over (The Rendering Pipeline).
- Real network for every request unless you intercept it, including third-party scripts, fonts and analytics beacons that have nothing to do with your assertions (Third-Party Scripts and the Supply Chain).
- Animation and transition work that the test then has to wait out, which is why disabling animation in the test environment removes both time and flake (Cheap and Expensive Animation).
- Video, trace and screenshot capture, which is the most valuable debugging artefact you can have and also a meaningful share of the suite's runtime and storage.
- Retries multiply all of the above, so a suite with a high flake rate is paying for its own flake in machine time as well as in trust.
The short list, and why it stays short
The question for every candidate flow is: if this broke in production and nobody noticed for an hour, what happens? For signup, login, checkout, payment and upload, the answer is revenue, support load or data loss. For a settings toggle, the answer is a bug report. The list is short because the answer is only severe for a few flows.
Write each one as a journey with the states a person actually passes through, including the ones people forget: the failure path, the retry, and what happens when they come back to the tab an hour later (Long-Lived Clients and Version Skew).
- 1Arrive authenticated
Session established through an API and injected, not by typing into the login form for the hundredth time.
fails by Logging in through the UI in every test, which triples the suite runtime and makes login a single point of failure for everything.
- 2Add to basket
Real interaction, then wait for the basket count to reach the expected value.
fails by Asserting immediately after the click, before the update has been applied.
- 3Go to checkout
Navigate and wait for the checkout heading; assert focus moved to the new content.
fails by Asserting on the URL only, which is true one frame before the page exists (History and Navigation).
- 4Enter payment details
Fill the fields, including the third-party frame if there is one.
fails by Racing the frame's own load; a cross-origin embed becomes interactive on its own schedule.
- 5Submit and wait for the outcome
Wait for a confirmation identified by role and text, not by a spinner disappearing.
fails by Treating the spinner vanishing as success, which is also what a failure looks like.
- 6Assert the durable effect
Reload, or check through the API, that the order exists.
fails by Asserting only on the confirmation screen, which can render from optimistic client state that was never persisted (Optimistic UI).
- 7Walk the failure path too
A declined card, an expired session, a network drop mid-submit.
fails by Being skipped, so the flow is proven only for the case that was already working.
Two steps here — injecting the session and asserting the durable effect — remove more flake and add more signal than any other change people make to these suites.
Where flake actually comes from
Flake has four homes: timing, animation, network and shared state. Each has a different fix, and none of the fixes is a retry. Sorting a flaky test into one of the four is most of the work of fixing it.
The honest framing is that a flaky test is evidence. Before you change the test, spend ten minutes asking whether a user could lose the same race — because if they can, the test has just done the most valuable thing it will ever do.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Assertion runs before the update lands | Fails on CI, passes locally | The test waited for a duration, or did not wait at all | Wait for a condition the application reaches — an element, a role, a value — and let the runner retry the query. |
| An element is moving when it is clicked | Click lands on the wrong element, or is reported as intercepted | A transition or entrance animation is still running | Disable animation in the test environment; it is the highest-return single setting in the whole suite. |
| A response arrives slower than usual | Intermittent timeout, or a stale value asserted | Two requests resolving out of order, or a timeout tuned to a fast day | Fix the ordering in the product — ignore responses older than the current request — and wait for the specific state, not for the network to be idle (Cancelling a Request Nobody Is Waiting For). |
| The suite is sharded or reordered | Failures that vanish when the test is run alone | Shared account, shared seed data, leftover storage or a persisted service worker | Give each test its own account, its own data and a clean storage state; assert nothing about the world outside its own fixtures. |
| Someone deploys to the shared environment | A whole run fails at an arbitrary point | The stack changed underneath the browser mid-run | Use an ephemeral or pinned environment per run so a failure is attributable to the change under test. |
| A test passes on retry | Green suite, unchanged product | A real race that the retry papers over | Treat it as a defect: reproduce with a throttled CPU or network, then fix the race (Heisenbugs: The Bug That Leaves When You Look at It in Concurrency). |
What a retry actually hides
Draw the interleaving that a flaky test found and the argument for retries collapses. Below, a token refresh and a form submission overlap. On a fast machine the refresh finishes first and everything works; on a loaded machine it does not, and the submission goes out with a credential that has just been rotated.
The test that failed here was not defective. It sampled an interleaving that a user will also sample, at a rate proportional to how slow their device and network are — which means the users who hit it are the ones already having the worst time. Retrying the test moves the failure from your CI dashboard to their session.
- User clicks Submit — The handler starts; the form reads the credential it currently holds.
- Token refresh in flight — Started earlier by a background check. Nothing in the submit path is waiting for it.
- Submit request sent — Carries the old credential, because the refresh has not resolved yet.
- Refresh resolves, credential rotated — The old credential is now invalid server-side.
- Submit rejected — On a fast machine the refresh landed before the click and this never happens.
- Test retried, passes — The second run wins the race. The product is unchanged and the user still loses it sometimes.
The fix is in the product, not the test: the submit path must await the in-flight refresh rather than read whatever credential happens to be current (Session Expiry and the Refresh Race).
How to build it
Most important first.
- Name the flows out loud and keep the list short: signup, login, checkout, payment, upload, and the two or three navigations the product cannot function without. Everything else is a candidate for a lower level.
- Give every test its own world: a fresh account created through an API rather than through the UI, its own data, and cleared storage. Setup through the UI is slow, flaky, and tests the same login a hundred times (What the Frontend Is Responsible For in Auth).
- Wait for conditions the application actually reaches — a heading, a row, a status message, a disabled button becoming enabled. Never wait for a duration; a fixed wait is both slower on a fast machine and flakier on a slow one.
- Disable animation and pin the clock in the test environment. Both reduce flake, and both are configuration rather than test code (Contrast, Colour and Motion covers the production version of the same switch).
- Decide deliberately what is real and what is stubbed. A payment provider's sandbox is realistic and slow; an intercepted request is fast and proves less. Both are legitimate — an accidental mixture is not.
- Treat a flaky test as a bug report against the product until proven otherwise. Reproduce it by slowing the network or throttling the CPU, and fix the race, not the wait (Network Failures Only the Client Can See).
- Run one critical flow entirely by keyboard. It is the highest-value accessibility test in the suite and it costs one extra spec (Keyboard Operability).
Keyboard, focus, semantics, announcement
A required field on every lesson in this domain, not a section added when there is room.
- A suite driven entirely by synthetic clicks proves nothing about whether the flow is operable. Add one spec per critical flow that reaches every control with Tab and activates it with Enter or Space (Keyboard Operability).
- End-to-end is the only level that can test focus across a navigation: after a route change, focus should move to the new content rather than staying on a control that no longer exists (Focus Management, Client-Side Routing).
- It is also where announcement of route changes can be verified — a live region or a focused heading telling a screen-reader user that the page changed at all (Live Regions and Announcement).
- Run automated rule checks at a few key points in each flow rather than on one page. The error state of a checkout form is where accessibility problems concentrate, and it only exists mid-flow (Accessibility Testing).
- Where the flow includes a third-party embed — a payment iframe, a captcha — record explicitly whether it is operable, because you inherit its accessibility and cannot fix it.
What can go wrong
- Retry-until-green. It converts a real race into a statistic and guarantees that the same race reaches production, where it has no retry.
- Shared accounts across parallel workers: one test signs out, three others fail, and none of the failures are about the code under test.
- Fixed sleeps as the waiting strategy. On a loaded CI machine they are too short; on a developer laptop they are pure waste.
- Tests that depend on order because an earlier test created the data a later one reads. It passes locally and fails the day the runner shards the suite.
- The mitigation failing: aggressive auto-quarantine of flaky tests silently removes coverage of exactly the flows most likely to be racy — which are the important ones.
- Asserting on text that is localised, formatted or time-dependent, so the suite fails in another timezone or after a copy change (Timezones and Locale Formatting).
- Pointing the suite at a shared staging environment that other people deploy to mid-run.
- A click landing during a re-render hits the element that was there a moment ago. A user hits this too; the test found it first (Reconciliation and Keys).
- A form submitted while a token refresh is in flight can be sent with the old credential. Intermittent in tests, intermittent in production, and the same bug (Session Expiry and the Refresh Race).
- Two requests started by one interaction resolve out of order, so the slower earlier response overwrites the newer one (Out-of-Order Responses).
- A navigation racing a redirect: the test asserts on a URL that was correct for one frame.
- Parallel workers writing to the same account or the same seeded row, so a test fails because a sibling changed the world underneath it.
- A service worker from a previous test serving a stale bundle to the next one (The Service Worker Lifecycle).
- These suites hold credentials. Test account passwords, API keys for seeding and provider sandbox secrets are real secrets living in CI configuration, and a captured trace or video can contain a session token in a request header (Secrets Management).
- Never point a destructive suite at production. Signup and checkout tests create real accounts, real orders and real charges; an environment you can wipe is part of the design (Development, Staging and Production in Cloud).
- End-to-end is the right level to prove that an unauthenticated user is actually redirected, that a session expiring mid-flow does not silently drop data, and that a logged-out tab cannot continue (Session Expiry and the Refresh Race, Auth Across Tabs).
- Recorded network fixtures are a disclosure risk twice over: they can carry tokens and they can carry customer data, and they live in the repository forever.
- "End-to-end tests are the realistic ones." They are realistic about integration only. Real users are on slower devices, worse networks, with assistive technology and an account that has three years of data in it.
- "Flake is just how browser tests are." Flake is a symptom with a cause: a wait on a duration, shared state, an animation, or a genuine race in the product. All four are diagnosable.
- "Retrying makes the suite reliable." It makes the suite *green*. Reliability moved to the user, who does not get a second attempt.
- "If we cover every flow end-to-end we can delete the lower levels." You would be trading a one-second failure that names a function for a six-minute failure that names a page.
- "The suite passed, so the release is safe." The suite exercised the flows on the list, in one browser, on one device class, with fresh data (Choosing the Test Level).
- "Page objects fix maintenance." They centralise selectors, which helps. They do nothing about isolation, waiting or races, which is where the cost actually is.
Measuring it, and what changes in the field
- Flake rate per test over a rolling window. It is the number that decides whether the suite is an asset or a tax, and it must be attributed per test, not per suite.
- Wall-clock time from push to signal, and where the suite runs — pull request, merge queue, or nightly. A suite that runs after merge is a monitoring tool, not a gate.
- Retry count as a first-class metric. A rising retry count is a rising number of unfixed races.
- For every escaped production incident in a critical flow, whether a test existed and what it did instead (Release Health, Frontend Error Tracking).
- On a loaded or shared CI machine, the application is slower than any developer has seen it, and every timing assumption in the suite is tested at once.
- On a throttled network, request ordering changes and races that never appear locally become reproducible — which makes throttling a debugging tool rather than only a hazard (Network Failures Only the Client Can See).
- Against a deploying backend, tests fail mid-run for reasons entirely outside the frontend. A pinned or ephemeral environment is what makes results attributable (Four Ways to Replace Running Code in Cloud).
- With a large seeded dataset, pages that were fast with ten rows are slow with ten thousand, and the suite starts timing out on pagination rather than on anything broken (Pagination From the Interface Backwards).
- This is the most expensive test level you own: machine time, maintenance time, debugging time and a permanent flake budget. The short list of flows is not timidity, it is what makes the suite survivable.
- Full isolation — a fresh account and fresh data per test — costs setup time and requires an API to create them. Sharing state is faster and is the single biggest source of order-dependent failures.
- Stubbing the network makes tests fast and deterministic while removing the integration you came for. Using the real thing is realistic and couples your suite to someone else's uptime.
- A hard flake policy keeps the suite trustworthy and will, sooner or later, delete a test that was catching something real. Quarantine with an owner is the compromise; both halves of it are work.
Where this applies
Frontend advice ages badly and fragments across engines. These labels say what each claim is specific to, and where a different browser, device or framework would differ.
- GENERALThe economics — high fidelity, high cost, flake as the dominant maintenance burden — follow from driving a real browser over a real stack and hold for every tool in this category.
- FRAMEWORK-SPECIFICRunners differ in ways that change how tests are written: Playwright and WebdriverIO drive multiple browser engines out of process and isolate storage per context, while Cypress runs the test in the page with a different async model, so waiting, isolation and multi-tab or multi-origin flows are not expressed the same way in each.
- DEVICE-SPECIFICA headless browser on a CI container is not a phone: no touch input by default, no real device pixel ratio, no thermal throttling and far more CPU. A green suite says nothing about the mid-range device most of your users are holding.
Where the depth lives
This domain teaches the browser-side mechanism and hands the rest off.
- — Testing & Reliability Engineering — flake as a measured phenomenon: quarantine policies, statistical detection of intermittent tests, and the argument for treating a flaky test as a defect in the system under test rather than in the test.
- — Distributed Systems — a browser, an API, a payment provider and a queue is a distributed system, and the ordering guarantees a test relies on are the ones that domain reasons about properly.