How to Tell a Flaky Test From a Real Regression

#How to Tell a Flaky Test From a Real Regression

A test fails.

The developer gets a notification.

The QA engineer opens the report.

The application looks fine.

The test is run again.

It passes.

So what happened?

Was there a real regression?

Was the environment unstable?

Did the test simply get unlucky?

Or is the test itself unreliable?

This is one of the most expensive problems in automated software testing because a flaky test doesn't simply create a red result.

It creates uncertainty.

And uncertainty is dangerous in QA.

When a team starts seeing failures that cannot be trusted, people gradually change their behavior.

They rerun tests.

They ignore occasional failures.

They mark tests as flaky.

They retry pipelines.

They wait for another build.

Eventually, a genuinely broken feature can fail alongside dozens of unreliable tests and nobody knows which failures deserve attention.

The result is worse than a failed test.

It is a loss of confidence in the testing system.

What is a flaky test?

A flaky test is a test that produces inconsistent results even though the underlying application behavior has not meaningfully changed.

For example:

Run 1 → Failed

Run 2 → Passed

Run 3 → Passed

Run 4 → Failed

Run 5 → Passed

If the same application state and test conditions produce inconsistent results, something is unstable.

That instability can come from many places.

It might be:

- Timing
- Network conditions
- Test data
- Browser behavior
- Environment instability
- Race conditions
- Asynchronous operations
- External services
- Shared state
- Poor test isolation
- Incorrect waits
- Randomized data
- Infrastructure failures

The important point is that a flaky test is not necessarily a bad test.

Sometimes the test is exposing a real weakness in the system.

For example, a race condition may genuinely exist in the application, but it only appears under certain timing conditions.

That makes classification difficult.

Why flaky tests are so dangerous

A normal failed test gives you a signal:

Something may be wrong.

A flaky test gives you an ambiguous signal:

Something may be wrong, or maybe nothing is wrong.

That distinction matters.

Imagine a CI pipeline containing 1,000 automated tests.

Suppose 20 tests fail.

If all 20 failures are trustworthy, the QA team can investigate them.

But what if 15 of those failures are historically flaky?

Now the team has to determine which failures represent actual product defects.

The testing system has become a filtering problem.

And that filtering consumes engineering time.

The first question should not be "Did it pass on retry?"

This is a common mistake.

A test fails.

Someone clicks rerun.

It passes.

The failure gets labeled "flaky."

Done.

But a passing retry doesn't automatically prove that the first failure was meaningless.

Consider a checkout test.

First execution:

Payment request fails.

Second execution:

Payment succeeds.

That could be a flaky test.

But it could also indicate:

- An intermittent backend failure
- A race condition
- A timeout
- A third-party payment problem
- A database issue
- A real application defect that occurs only under certain conditions

The retry tells you that the behavior is inconsistent.

It doesn't necessarily tell you why.

Look at the failure pattern

One of the strongest ways to investigate flaky behavior is to examine historical patterns.

Suppose a test has produced these results over the last 50 executions:

42 passed

5 failed because of timeout

3 failed because of an element not being available

That pattern deserves investigation.

Now compare it with another test:

49 passed

1 failed because a button was genuinely missing

The second test may be much more reliable.

Historical data provides context that a single test execution cannot.

For each test, teams should ideally know:

- Total executions
- Total failures
- Failure rate
- Recent failure rate
- Failure reasons
- Average execution time
- Retry rate
- Environment associated with failures
- Browser associated with failures
- Whether failures were later confirmed as real bugs

This turns "flaky" from a subjective label into something measurable.

A useful concept: failure consistency

Ask:

Does the test fail in a consistent way?

Imagine a login test that repeatedly fails with:

"Submit button not found."

That's different from a test failing with completely unrelated errors every time.

Consistent failures may indicate a genuine application or test problem.

Inconsistent failures may point toward environmental instability or timing issues.

For example:

Run 1:
Element not found

Run 2:
Timeout

Run 3:
Network error

Run 4:
Browser disconnected

Run 5:
Passed

This pattern suggests that something broader may be unstable.

The test itself may not be the only suspect.

Check whether the failure is reproducible

Reproducibility is one of the most useful signals in debugging.

If a test fails once and passes 20 consecutive times afterward, the failure requires investigation but may be difficult to reproduce.

If it fails every time under the same conditions, it is much more likely to represent a deterministic problem.

Consider:

Test A:

Fail → Pass → Pass → Pass → Pass

Test B:

Fail → Fail → Fail → Fail → Fail

Test B should receive immediate attention.

Test A requires investigation into intermittent conditions.

Neither should automatically be dismissed.

Compare the application state

A screenshot can sometimes tell you more than the test error message.

Suppose the test reports:

"Element not found."

The screenshot shows that the entire page is still loading.

That's very different from a screenshot showing a completely rendered page where the expected element is genuinely absent.

Another example:

Expected:

"Order confirmed"

Actual:

"Payment failed"

The test may have exposed a real product issue.

Visual evidence gives context to otherwise ambiguous errors.

This is especially useful for browser-based testing because the state of the interface can change quickly.

Timing is one of the biggest sources of flaky browser tests

Modern web applications are highly asynchronous.

A page can appear visually loaded while data is still being fetched.

A button can exist in the DOM but not yet be interactive.

A network request can finish after the test expects it to.

A modal can appear milliseconds later.

These situations create timing-sensitive tests.

For example:

1. Open checkout.
2. Click "Pay."
3. Immediately search for "Payment successful."

The test may fail if the response takes longer than expected.

Run the same test again.

The server responds faster.

The test passes.

That looks like randomness.

But the real issue may be that the test makes an unsafe timing assumption.

Don't solve every flaky test by adding a longer wait

This is another common automation mistake.

The test fails because an element isn't ready.

The team adds:

wait 5 seconds

It fails again.

They change it to:

wait 10 seconds

Eventually the test passes most of the time.

But the underlying synchronization problem hasn't been solved.

Now the suite is slower.

And the test may still fail under unusual conditions.

A better approach is to wait for a meaningful condition.

For example:

- Element becomes visible
- Element becomes enabled
- Network request completes
- Specific response arrives
- Loading indicator disappears
- Expected page state appears

The objective is not:

"Wait longer."

It is:

"Wait for the application state the test actually depends on."

Test data can create fake failures

Sometimes the application is fine.

The test data isn't.

Imagine a test that attempts to create a user with:

test@example.com

The first run succeeds.

The second run fails because the email already exists.

The application hasn't regressed.

The test has created a state dependency.

Other examples include:

- Expired accounts
- Used coupon codes
- Existing orders
- Deleted records
- Exhausted inventory
- Changed permissions
- Shared database records

This is why test isolation matters.

A test should ideally be able to run without depending unpredictably on what another test did earlier.

Environment failures can look like application failures

Consider a test that expects an API response within two seconds.

The application normally responds in 300 milliseconds.

But the test environment is experiencing network congestion.

The response takes four seconds.

The test fails.

That failure doesn't necessarily mean the application's normal behavior is broken.

It means the test environment violated the expected condition.

This is why good test reports should capture environmental information.

Useful signals include:

- Browser
- Operating system
- Environment
- Network errors
- HTTP status codes
- Response times
- Console errors
- Screenshots
- Video
- Logs
- Timestamp

Without this context, QA teams are forced to investigate failures almost blindly.

External services make classification harder

Modern applications depend on other systems.

Payments.

Authentication providers.

Email services.

Cloud APIs.

Maps.

Analytics.

Third-party databases.

A test can fail because an external dependency temporarily fails.

That doesn't automatically mean your application is defective.

However, it also doesn't mean the failure should simply be ignored.

If the application cannot handle the external failure gracefully, that can itself be a real defect.

For example:

Payment provider returns an error.

Application displays:

"Something went wrong."

That may be acceptable.

But if the application:

- Charges the customer twice
- Creates an order without payment
- Loses the cart
- Shows a misleading success message

then the external failure has exposed a genuine application problem.

The distinction is important.

Separate test failure from product failure

A useful QA investigation asks two different questions:

Question 1

"Did the test fail?"

This is objective.

Question 2

"Did the product behave incorrectly?"

This requires analysis.

A test failure is evidence.

It isn't automatically a bug.

This distinction becomes increasingly important as automated testing scales.

The more tests you run, the more important failure interpretation becomes.

A practical failure classification system

Teams can classify failures into a few categories.

Category 1: Confirmed product defect

The application behaves incorrectly and the failure can be reproduced or supported by strong evidence.

Action:

Create a bug.

Category 2: Test defect

The test itself contains an incorrect assumption or implementation.

Action:

Fix the test.

Category 3: Environment failure

The infrastructure, browser, network, database, or environment caused the failure.

Action:

Fix or investigate the environment.

Category 4: External dependency failure

A third-party service caused the failure.

Action:

Determine whether the application handled the dependency failure correctly.

Category 5: Intermittent behavior

The failure cannot yet be reliably reproduced and requires additional evidence.

Action:

Track it and investigate rather than immediately ignoring it.

This simple classification can dramatically improve how teams handle automation failures.

What AI can contribute

AI is particularly interesting here because failure investigation is not simply a pass/fail problem.

A useful AI testing system can compare:

Current failure

against

Historical failures.

It can look at:

- Screenshots
- Error messages
- DOM state
- Browser console
- Network failures
- Execution time
- Previous test results
- Similar failures
- Related test cases

Then it can help answer:

"Have we seen this failure before?"

"Does this look like a known flaky pattern?"

"Did similar tests pass?"

"Did the failure occur only in one browser?"

"Was the backend returning errors?"

"Does the screenshot indicate that the expected element actually existed?"

This doesn't mean AI can magically determine the truth every time.

It means AI can reduce the amount of manual investigation required.

Imagine a better failure report

Instead of:

TEST FAILED

Element not found.

A richer report could say:

Test:
Checkout payment

Result:
Failed

Confidence:
Medium

Potential cause:
Payment confirmation element was not visible within the expected timeframe.

Evidence:

- Checkout page loaded successfully
- Payment request returned HTTP 200
- Confirmation UI appeared 4.2 seconds after request
- Test timeout was 3 seconds
- Same test passed in the previous 18 executions
- Two other checkout tests also experienced increased response times

Suggested classification:

Potential timing/performance issue rather than a confirmed functional regression.

That is a much more useful result for a QA engineer.

The engineer can then investigate the evidence instead of spending several minutes simply figuring out what happened.

Flakiness should be measured separately from bugs

One mistake teams make is mixing everything into one failure metric.

For example:

"20% of our tests are failing."

That number isn't enough.

You want to know:

20% failed.

But:

12% were environment failures.

4% were known flaky tests.

2% were test implementation problems.

2% were confirmed product regressions.

Now the situation looks completely different.

The product may not actually be experiencing a 20% regression rate.

The testing system may be experiencing a 20% failure rate.

Those are very different problems.

Create a quarantine strategy

Some tests are so unreliable that they can disrupt the entire pipeline.

Instead of allowing them to repeatedly block releases, teams can place them into a quarantine state.

A quarantined test should not disappear.

It should remain visible.

For example:

Flaky Test

Status:
Quarantined

Failure rate:
18%

Last confirmed product defect:
42 executions ago

Primary suspected cause:
Timing

Owner:
Checkout QA

Next review:
Scheduled

This is much better than simply deleting the test or ignoring its failures.

But don't create a permanent "flaky" graveyard

Quarantine should not become a place where broken tests go to die.

Every quarantined test should have an owner and a reason.

Otherwise, the team eventually ends up with hundreds of ignored tests.

A good process is:

Detect

↓

Classify

↓

Quarantine if necessary

↓

Investigate

↓

Fix

↓

Return to normal regression

The objective is to reduce flakiness over time.

A simple decision tree for QA engineers

When an automated test fails, ask:

1. Can I reproduce the failure?

Yes → investigate the product or test.

No → continue.

2. Did similar tests fail?

Yes → investigate a broader system issue.

No → continue.

3. Did the environment show unusual behavior?

Yes → investigate infrastructure/environment.

No → continue.

4. Did the test fail previously in the same way?

Yes → inspect historical evidence.

No → continue.

5. Does the screenshot/log/network evidence show incorrect product behavior?

Yes → potential real regression.

No → investigate test reliability.

This doesn't eliminate ambiguity.

It gives the team a consistent way to handle it.

The goal isn't zero flaky tests overnight

Large systems rarely become perfectly stable immediately.

A better goal is to make flakiness visible and measurable.

Track:

- Flaky test percentage
- Failure frequency
- Mean time to investigate
- Mean time to fix
- Quarantined tests
- Repeated failures
- False-positive failures
- Confirmed regressions

Once you have those numbers, you can improve the system systematically.

What a mature AI testing workflow could look like

A modern testing platform could continuously learn from test execution.

Test runs.

↓

Capture results and evidence.

↓

Compare against historical executions.

↓

Detect unusual failure patterns.

↓

Classify likely causes.

↓

Identify similar previous failures.

↓

Prioritize failures requiring human attention.

↓

Learn from the final QA classification.

Over time, the system becomes better at distinguishing:

"This looks like a real regression."

from:

"This looks like a known flaky pattern."

That is a much more valuable form of AI assistance than simply generating another test script.

RoboQA's role in this direction

AI browser testing should not stop at:

"Click this button and tell me whether it worked."

The more interesting problem is what happens after the test fails.

A useful QA system should help answer:

What happened?

Where did it happen?

Can we reproduce it?

Has it happened before?

Is this similar to another failure?

What evidence supports the failure?

Does the behavior look like a product regression, test issue, or environmental problem?

This is where screenshots, browser behavior, test history, and AI analysis can work together.

The goal isn't to replace the QA engineer.

The goal is to give the QA engineer better evidence.

Final takeaway

A failed automated test is not automatically a bug.

It is a signal that needs interpretation.

The difference between a mature QA organization and an immature automation setup is often not how many tests they have.

It is how effectively they interpret failures.

When a test fails, look at:

- Reproducibility
- Historical behavior
- Failure consistency
- Timing
- Test data
- Environment
- External dependencies
- Screenshots
- Logs
- Network activity
- Related test failures

Then classify the result.

Is it:

A product defect?

A test defect?

An environment problem?

An external dependency?

Or an intermittent failure that needs more evidence?

The more automated your testing becomes, the more important this distinction becomes.

Because eventually the problem isn't:

"We don't have enough tests."

It becomes:

"We have thousands of test results. Which ones should we believe?"

That's the next challenge of software testing.

And AI can help — not by blindly declaring every failure a bug, but by helping QA teams turn noisy test results into evidence they can actually act on.