A flaky suite is worse than no suite Once a red build might mean nothing, people stop reading red builds, and the suite you paid for is now decoration. Flakiness is not bad luck; it comes from a short list of habits, and each has a fix. These are the rules every suite Apex Automation Team builds has to follow.
1. Locate elements the way a user identifies them
Selectors tied to CSS classes, deep DOM paths or positional indexes break on the next redesign even though nothing functional changed. Prefer the accessible identity of the element: its role and name, its label, its placeholder, its visible text. Where a control has no stable accessible identity, add a dedicated test attribute in the application and target that. A test attribute is a contract with the test suite, and it survives styling changes.
2. Never sleep for a fixed number of seconds
A fixed wait is a bet that the machine will be as fast today as it was yesterday. On a loaded CI runner it loses. Modern frameworks already wait for an element to exist, be visible, be stable and be able to receive events before acting, so the fix is usually to delete the sleep. Where something genuinely needs waiting for, wait for the condition: the request finished, the URL changed, the spinner disappeared, the row count reached what it should be.
3. Assert on state, not on a moment
Use assertions that retry until they pass or time out, rather than reading a value once and comparing it. Reading the text of an element the instant after a click is a race. Asserting that the element eventually contains that text is not.
4. Every test creates its own world
Tests that share an account, a cart or a database row will pass alone and fail in parallel, and the failure will look random. Each test should create its data, act, and leave nothing behind that another test depends on. Never rely on execution order. Log in through the API and reuse the stored session instead of driving the login form three hundred times.
5. Control what you do not own
Third-party widgets, payment sandboxes, analytics scripts and advertising all fail sometimes, and none of it is your product's fault. Intercept those calls and serve a fixed response, so a slow external service is not reported as your bug. Pin dates and random values too: a test that breaks on the last day of the month is a time bomb.
6. Retry once, and record everything about the retry
A single automatic retry absorbs genuine infrastructure blips. More than that hides real defects. The important part is the evidence: capture a trace on the first retry, a video on failure and a screenshot at the moment of the error, so a human can see what happened instead of re-running it locally and hoping.
Any test that needed a retry is still a signal. Track them. A test that flakes weekly is a bug in the test or in the product, and it gets fixed or deleted, never left to rot.
7. Keep the pre-merge run short
Shard the suite across parallel workers and keep the set that runs on every pull request under ten minutes. Past that, people start merging without waiting, and the suite might as well not exist.
8. Quarantine, do not disable quietly
When a test proves unreliable, move it to a named quarantine set that still runs and still reports, with an owner and a date. Commenting it out is how suites silently shrink until the day nobody trusts them at all.
We build suites in Playwright, Puppeteer or Cypress, documented and handed over in your repository. See software testing and QA or how to build an automated regression suite.
Quick checklist
- Locate elements by role, label or a dedicated test attribute
- Delete every fixed sleep and wait for a condition instead
- Use assertions that retry rather than reading a value once
- Give each test its own data and never depend on order
- Log in through the API and reuse the session
- Intercept third-party calls and pin dates and random values
- Allow one retry, capture a trace and video, and track every flake
- Keep the pull-request run under ten minutes with parallel workers
Frequently asked
Is a retry hiding a real bug?
It can be, which is why a retry is recorded rather than forgiven. One retry absorbs infrastructure blips; a test that needs retries regularly is investigated and then fixed or deleted.
Why not just add longer waits?
Longer waits make the suite slow without making it reliable, because the race condition is still there. Waiting for the actual condition is both faster and stable.
How long should the suite take to run?
The set on every pull request should finish in under ten minutes or developers stop waiting for it. The full cross-browser run belongs nightly, and the exhaustive run before a release.