Flaky Tests: Why Automated Tests Fail Randomly and How to Fix Them
There is a particular kind of failure that every automation engineer eventually learns to hate.
You run the test.
It fails.
You run it again.
It passes.
You run it again.
It fails.
You run it a fourth time.
It passes.
Nothing changed in the application.
Nothing changed in the code.
Nothing changed in the test.
Yet the result keeps changing.
This is the world of flaky tests.
Flaky tests are automated tests that produce inconsistent results even though the software under test has not meaningfully changed.
They are one of the biggest problems in modern test automation because they damage something that automation depends on:
Trust.
If a test fails, the team should immediately want to know why.
But when engineers know that a particular test fails randomly, the reaction can become:
"Probably flaky."
That sentence can be dangerous.
A real product defect can be dismissed as test instability.
A broken build can be ignored.
A regression can be overlooked.
Eventually, the automated test suite becomes something the team works around rather than something the team trusts.
This is why flaky test management should be treated as an important part of software quality engineering.
The goal isn't simply to make every test pass.
The goal is to make test results reliable enough to support engineering decisions.
---
What Is a Flaky Test?
A flaky test is an automated test that can produce different results when executed under essentially the same conditions.
For example:
Run 1 → PASS
Run 2 → PASS
Run 3 → FAIL
Run 4 → PASS
Run 5 → FAIL
If the application hasn't meaningfully changed, this inconsistent behavior indicates possible test flakiness.
The test may not be completely random.
It could fail only under certain timing conditions.
It might fail only on a particular browser.
It could fail when the CI environment is under heavy load.
It might fail when another test modifies shared data.
It might fail when an API response takes longer than usual.
This makes flaky tests particularly difficult to diagnose.
---
Flaky Test vs Real Product Bug
This distinction is critical.
Suppose a checkout test fails.
The first question should not be:
"Is this flaky?"
The first question should be:
What evidence tells us why it failed?
A failure can come from several sources.
Product Defect
The application behaves incorrectly.
Automation Defect
The test itself is incorrect.
Environment Problem
The test environment is unstable.
Test Data Problem
The required data is missing or inconsistent.
External Dependency
A third-party service behaves unexpectedly.
Timing Problem
The test interacts with the application before it is ready.
A good QA process investigates the failure instead of immediately labeling it flaky.
---
Why Flaky Tests Are Dangerous
Flaky tests create noise.
Imagine a CI pipeline with 1,000 tests.
Normally:
990 pass
10 fail
Now suppose 7 of those 10 failures are flaky.
The team investigates.
Three are real product defects.
Seven are test problems.
If this happens repeatedly, engineers become less responsive to failures.
That is known as a loss of trust in the test suite.
The worst possible outcome is:
A real failure gets ignored because the team assumes the test is flaky.
That makes test reliability a quality problem, not merely an automation problem.
---
The Most Common Causes of Flaky Tests
Flakiness can have many causes.
Some of the most common include:
- Timing issues
- Race conditions
- Unstable selectors
- Network instability
- Shared test data
- Environment instability
- External services
- Random data
- Test order dependencies
- Browser differences
- Asynchronous application behavior
- Poor cleanup
- Resource constraints
Let's examine these individually.
---
1. Timing Problems
Timing is one of the most common causes of flaky browser tests.
Imagine a page loads.
The test immediately looks for a button.
The browser hasn't finished rendering the component.
The test fails.
Run it again.
This time the page loads slightly faster.
The test passes.
The application didn't necessarily change.
The timing did.
This creates classic test flakiness.
---
Fixed Sleeps Are Usually Not the Best Solution
A common response is:
"Let's wait five seconds."
So the test becomes:
Open page.
Wait five seconds.
Click button.
This might appear to solve the problem.
But it creates another issue.
What if the page loads in one second?
The test unnecessarily waits four seconds.
What if the page takes seven seconds?
The test still fails.
Fixed waits don't understand application state.
A better approach is usually to wait for a meaningful condition.
For example:
Wait until the button is visible.
Wait until the API response completes.
Wait until the loading indicator disappears.
Wait until navigation finishes.
The exact implementation depends on the automation framework.
The principle is:
Wait for state, not arbitrary time.
---
2. Race Conditions
A race condition can occur when two processes compete or when the test interacts with the application before another operation has completed.
For example:
The user clicks Save.
↓
API request starts.
↓
The test immediately checks the database.
↓
Database update hasn't completed.
↓
Test fails.
Run again.
↓
API finishes faster.
↓
Test passes.
This is a classic synchronization problem.
The solution is not necessarily "wait longer."
The solution is to understand the application's state transition and synchronize the test appropriately.
---
3. Unstable Selectors
Browser automation often needs to identify elements.
Examples include:
- IDs
- CSS selectors
- XPath
- Text
- Roles
- Attributes
If the selector depends on unstable DOM structure, tests can become fragile.
For example, a selector based on a deeply nested hierarchy may break after a harmless UI refactor.
The application still works.
The test fails.
This isn't necessarily classic timing-based flakiness, but fragile element identification can contribute to unreliable automation.
Prefer selectors that represent stable application intent where possible.
---
4. Network Instability
Web applications depend heavily on networks.
A browser test may interact with:
- APIs
- Authentication services
- Payment systems
- Content delivery networks
- Third-party integrations
A temporary network delay can cause a test to fail.
Run again.
The network responds normally.
The test passes.
This can create flaky results.
Teams should therefore distinguish:
Application failures
from:
Infrastructure failures
and:
External dependency failures.
---
5. Shared Test Data
Test data is another major source of instability.
Imagine two tests use the same account.
Test A changes the user's profile.
Test B expects the original profile.
Depending on execution order, Test B may pass or fail.
This creates a hidden dependency.
The solution is to isolate test data where practical.
Each test should create or obtain the state it needs rather than assuming another test has prepared it correctly.
---
6. Test Order Dependency
Tests should ideally be independent.
Consider:
Test A → creates customer
Test B → modifies customer
Test C → verifies customer
If Test A fails, B and C may fail too.
Worse, if tests execute in a different order, results may change.
A test suite that passes only when executed in one specific order is a warning sign.
Tests should be designed to minimize dependencies on previous test execution.
---
7. Environment Instability
Sometimes the application isn't the problem.
The test environment is.
For example:
- CPU is overloaded
- Memory is exhausted
- Database is slow
- Containers restart
- Services are unavailable
- Network latency increases
Tests can fail even though the application would behave correctly in a healthy environment.
This is why CI infrastructure needs monitoring too.
A test failure without environment context can be difficult to interpret.
---
8. External Dependencies
Third-party services can create flaky tests.
Imagine a test depends on:
- Payment provider
- Email service
- Maps API
- Authentication provider
- External analytics service
If that external system is temporarily unavailable, your test may fail.
That doesn't necessarily mean your application is broken.
Where appropriate, teams can use:
- Mocks
- Stubs
- Test environments
- Sandboxed integrations
for deterministic testing.
But critical end-to-end integrations should still be tested realistically at appropriate stages.
---
9. Random Test Data
Randomness can make tests difficult to reproduce.
Suppose an automated test generates:
Random username
Random product
Random address
Random date
Most runs pass.
One run fails.
Now the engineer needs to know exactly what data caused the failure.
Random data can be valuable for discovering edge cases.
But when randomness is used, the test should preserve enough information to reproduce the failing scenario.
A useful principle is:
Randomness without reproducibility creates debugging problems.
---
10. Asynchronous Applications
Modern web applications frequently perform work asynchronously.
For example:
User clicks button.
↓
Frontend sends API request.
↓
Backend processes request.
↓
Frontend receives response.
↓
UI updates.
A test that assumes the UI changes immediately may become flaky.
Good automation needs to understand these transitions.
This is particularly important for:
- React applications
- Vue applications
- Angular applications
- Single-page applications
- AJAX-heavy interfaces
- Real-time interfaces
---
11. Browser Differences
A test can pass in one browser and fail in another.
For example:
Chrome → PASS
Firefox → PASS
Safari → FAIL
That doesn't automatically mean the test is flaky.
It may indicate a browser-specific product defect.
But inconsistent browser behavior can also expose weaknesses in the test.
Therefore, browser-specific failures need investigation.
Don't automatically label them flaky.
---
12. Poor Cleanup
Tests often create state.
For example:
- Users
- Orders
- Files
- Database records
- Sessions
- Cookies
If the test doesn't clean up properly, the next execution may behave differently.
Run one:
PASS.
Run again:
FAIL.
Reset environment:
PASS.
This is a strong indication that state management needs attention.
---
How to Detect Flaky Tests
The simplest method is repeated execution.
Suppose a test is executed 100 times.
Results:
PASS = 91
FAIL = 9
If the failures occur without meaningful product changes, investigate for flakiness.
However, repetition alone isn't enough.
You also need to capture context.
Useful information includes:
- Timestamp
- Browser
- Operating system
- Environment
- Test data
- Logs
- Screenshots
- Video
- Network information
- Application version
- Commit
- Failure message
This makes diagnosis much easier.
---
Flakiness Rate
Teams can calculate a basic flakiness rate.
For example:
Flakiness Rate = Inconsistent Runs / Total Runs
Suppose:
100 executions
8 inconsistent failures
Flakiness rate = 8%
The exact calculation can vary depending on how your organization defines a flaky failure.
The important part is tracking it consistently.
---
Don't Treat Every Failure as Flakiness
This deserves emphasis.
If a test fails three times in a row after a code change, don't immediately call it flaky.
The application may genuinely be broken.
A practical investigation process is:
1. Reproduce the failure.
2. Review logs.
3. Check screenshots.
4. Check network activity.
5. Compare with previous runs.
6. Check environment health.
7. Re-run under controlled conditions.
8. Determine the most likely cause.
Only then classify it.
---
A Useful Failure Classification
You can classify automated failures into:
Product Defect
Application behavior is incorrect.
Test Defect
Automation implementation is incorrect.
Infrastructure Failure
Environment caused failure.
Data Failure
Required test state was invalid.
External Dependency Failure
Third-party system caused failure.
Flaky / Intermittent
Same test produces inconsistent results under comparable conditions.
This classification helps teams prioritize the right response.
---
The Flaky Test Lifecycle
A useful workflow is:
Test fails
↓
Investigate
↓
Classify
↓
Confirm flakiness
↓
Create issue
↓
Quarantine if necessary
↓
Fix
↓
Run repeatedly
↓
Restore to normal suite
This prevents known flaky tests from silently damaging confidence.
---
Should Flaky Tests Be Removed?
Not always.
If a test protects a critical workflow, deleting it may create a larger problem.
Instead:
Fix it if possible.
If it cannot immediately be fixed, consider temporarily quarantining it while keeping visibility.
But quarantine should not become permanent.
A test that remains quarantined for six months is effectively a deleted test.
The team should periodically review quarantined tests.
---
What Is a Quarantined Test?
A quarantined test is usually isolated from the main release-blocking test suite because it is known to be unreliable.
For example:
Main regression suite:
999 reliable tests
Quarantine:
12 flaky tests
The 12 tests can continue running separately while the team investigates them.
This prevents known instability from blocking every build.
But quarantine needs ownership.
Every quarantined test should have:
- Reason
- Owner
- Date
- Tracking issue
- Expected resolution
---
Don't Let Quarantine Become a Graveyard
This is a common problem.
A test becomes flaky.
Someone moves it to quarantine.
Everyone forgets about it.
Months later:
20 tests are quarantined.
Then 50.
Then 100.
The main suite looks healthy because the unreliable tests have been removed.
But actual coverage is shrinking.
Quarantine should therefore be treated as temporary debt.
---
The Cost of Flaky Tests
Flakiness creates several hidden costs.
Engineering Time
Someone must investigate failures.
Pipeline Delays
Teams rerun jobs.
Lost Confidence
Engineers stop trusting automation.
Escaped Defects
Real failures may be ignored.
Maintenance Debt
Flaky tests require repeated attention.
Slower Releases
Teams wait for uncertain results.
This is why flaky test reduction can have a surprisingly large impact on engineering productivity.
---
Flaky Tests and CI/CD
CI/CD makes flakiness particularly painful.
Imagine every pull request runs:
2,000 tests.
If 20 are flaky, developers may repeatedly restart pipelines.
This creates:
Commit
↓
Pipeline fails
↓
Rerun
↓
Pipeline passes
↓
Merge
This process gives a false sense of confidence.
The second run didn't necessarily prove the code was correct.
It only proved the test passed once.
---
Retry Policies Need Care
Retries can be useful.
For example, a transient infrastructure failure may justify one retry.
But retries should not become:
Failure → retry until green.
That turns testing into a search for a passing result.
A better approach is to record:
First attempt → FAIL
Retry → PASS
Then investigate why.
The pipeline should preserve the original failure information.
---
Test Reliability Should Be a Metric
Teams often track:
- Pass rate
- Coverage
- Execution time
They should also track:
Test reliability.
For example:
- Flaky tests per 1,000 tests
- Flaky failure percentage
- Average time to repair flaky tests
- Number of quarantined tests
- Repeated failure rate
These metrics reveal whether automation is becoming healthier.
---
A Green Pipeline Isn't Always Healthy
Consider two pipelines.
Pipeline A
1,000 tests
995 pass
5 fail
All five failures are genuine defects.
Pipeline B
1,000 tests
1,000 pass
But 50 tests are flaky and happened to pass today.
Which pipeline provides more confidence?
Pipeline A may actually be more trustworthy.
A green pipeline is useful.
But green does not automatically mean reliable.
---
Flaky Tests and Regression Testing
Regression testing depends heavily on reliability.
If regression results can't be trusted, the suite loses much of its purpose.
This is why flaky test management should be part of regression strategy.
A regression test should ideally have:
- Clear purpose
- Stable data
- Reliable environment
- Meaningful assertions
- Reproducible failures
---
Flaky Tests and Test Prioritization
This connects directly with test prioritization.
Suppose you have:
Test A
Critical payment workflow.
Reliable.
Test B
Low-risk profile test.
Flaky.
Test A should clearly receive high priority.
But what about:
Test C
Critical authentication workflow.
Flaky.
You shouldn't simply lower its priority because it is flaky.
Instead:
Run it because the risk is high.
Then prioritize fixing the flakiness because the result is important.
This is why priority and reliability should be separate dimensions.
---
AI Can Potentially Help Detect Flakiness
This is one area where AI-assisted QA can become useful.
A system can analyze large amounts of test history.
For example:
Test X:
100 executions
94 passes
6 failures
Failures occurred across:
- different browsers
- different times
- different commits
No corresponding application defects were recorded.
That could be a signal worth investigating.
AI could potentially identify patterns humans might overlook across thousands of test executions.
---
AI Shouldn't Automatically Declare a Test Flaky
Again, caution is important.
A model might see:
Test fails often.
Therefore:
"Flaky."
But what if the application actually has a serious intermittent defect?
For example:
A payment service fails only under high concurrency.
The test passes most of the time.
That doesn't make it a useless flaky test.
It might be detecting a real intermittent production problem.
This is why AI classification needs context.
---
Intermittent Bugs Are Not the Same as Flaky Tests
This distinction is extremely important.
Flaky Test
The test result is unreliable because the test or environment is unstable.
Intermittent Product Bug
The product itself behaves incorrectly only under certain conditions.
Both can produce:
PASS
FAIL
PASS
FAIL
But the causes are completely different.
A mature QA system should try to distinguish them.
---
AI Could Help Correlate Failures
Imagine a test fails.
AI examines:
- Application logs
- Network failures
- Browser console
- Screenshots
- Test history
- Commit changes
- Environment health
It may suggest:
"Failure correlates with API timeout."
Or:
"Failure began immediately after the latest authentication change."
Or:
"Failure occurs only in Safari."
That doesn't prove the root cause.
But it can reduce investigation time.
---
Test Failure Analysis Is a Major Opportunity
Generating tests is only one part of automation.
A large amount of QA time is spent after a test fails.
What happened?
Why?
Is it reproducible?
Who owns it?
Is it a product defect?
Is it automation?
Is the environment broken?
This is where intelligent failure analysis could potentially provide significant value.
---
How to Reduce Flaky Tests
A practical strategy includes:
Use Reliable Synchronization
Wait for meaningful conditions.
Stabilize Selectors
Prefer robust element identification.
Isolate Test Data
Avoid unnecessary shared state.
Make Tests Independent
Don't depend on previous tests.
Control Environments
Keep CI environments stable.
Mock External Dependencies Where Appropriate
Reduce unnecessary uncertainty.
Capture Evidence
Screenshots, logs, video and network data help diagnosis.
Track Test History
Look for patterns.
Review Quarantined Tests
Don't allow permanent test debt.
---
Example: Fixing a Flaky Login Test
Suppose:
Login test fails 10% of the time.
Investigation shows:
The test enters credentials immediately after page navigation.
Sometimes the login form hasn't finished loading.
Instead of:
Wait 3 seconds.
Use a meaningful condition:
Wait until the login form is ready.
Then:
Enter credentials.
Click Login.
Wait for dashboard.
Verify dashboard.
The test becomes synchronized with application state.
---
Example: Fixing Shared Data Flakiness
Suppose:
Test A creates user "test@example.com."
Test B also expects "test@example.com."
When tests run in parallel, one may modify or delete the user.
The solution might be:
Generate isolated test accounts.
Or:
Create required state independently.
Or:
Use controlled fixtures.
The goal is to remove hidden dependencies.
---
Example: Fixing API-Dependent Flakiness
Suppose:
A UI test depends on a slow third-party API.
Sometimes the API takes 2 seconds.
Sometimes 10 seconds.
Sometimes it is unavailable.
If the test's purpose is to validate your UI logic rather than the third-party service, mocking may make sense.
If the purpose is to validate the actual integration, a real integration test should exist separately.
This creates clearer test boundaries.
---
Don't Make Every Test End-to-End
End-to-end tests are valuable.
But if every test is end-to-end, the suite can become:
- Slow
- Fragile
- Expensive
- Difficult to debug
A healthy testing strategy uses different levels.
Unit
Fast feedback.
API / Integration
Service behavior.
Browser
Important user journeys.
Exploratory
Human investigation.
This reduces pressure on browser automation.
---
The Testing Pyramid Still Matters
The testing pyramid encourages more tests at lower levels and fewer expensive end-to-end tests.
A simplified model:
End-to-End
/ \
Browser UI
/ \
API / Integration
/ \
Unit Tests
The exact shape doesn't need to be perfect.
The principle is:
Use the fastest reliable test level capable of detecting the risk.
Don't use a browser test when a unit or API test can provide the same valuable evidence.
---
Flaky Tests Should Influence Architecture
Sometimes flakiness exposes deeper problems.
For example:
A test frequently fails because the application has unclear state transitions.
That may indicate an application design issue.
Another test frequently fails because services communicate unpredictably.
That may indicate architectural instability.
Therefore, flaky tests can sometimes be useful signals about the system itself.
Don't automatically assume every flaky test is "just an automation problem."
---
Building a Reliable Regression Suite
A reliable regression suite should aim for:
Clear Test Purpose
Why does this test exist?
Stable Data
What state does it require?
Reliable Environment
Where does it execute?
Meaningful Assertions
What does success actually mean?
Reproducible Failures
Can engineers understand what happened?
Controlled Dependencies
What external services are involved?
Ownership
Who maintains the test?
This turns automation into an engineering asset rather than a collection of scripts.
---
A Practical Flaky Test Management Process
Here is a simple process teams can implement.
Step 1
Track test execution history.
Step 2
Identify repeated inconsistent failures.
Step 3
Investigate the failure.
Step 4
Classify the root cause.
Step 5
Fix the test, environment or application.
Step 6
Run the test repeatedly.
Step 7
Measure reliability.
Step 8
Return the test to the main suite.
This creates continuous improvement.
---
A Simple Reliability Classification
You can classify tests as:
Stable
Consistently produces expected results.
Watch
Occasional unexplained failures.
Flaky
Repeated inconsistent failures.
Quarantined
Known unreliable and temporarily removed from release blocking.
Disabled
No longer relevant or currently unusable.
This provides visibility into test health.
---
What QA Leaders Should Monitor
A useful QA dashboard could include:
Total automated tests
Critical tests
Flaky tests
Quarantined tests
Average execution time
Failure rate
Mean time to investigate
Mean time to repair
Escaped defects
These metrics provide a more realistic view of automation health.
---
The Goal Isn't Zero Failures
This may sound strange.
But a healthy test suite should not aim for:
"Zero failures."
If the application has a real defect, the test should fail.
The goal is:
Failures should be meaningful.
A test failure should provide information.
If the team can trust:
FAIL = something needs investigation
then automation is doing its job.
If the team thinks:
FAIL = probably flaky
then automation has lost much of its value.
---
How AI Changes the Flaky Test Problem
AI may eventually make test creation much easier.
But that makes reliability even more important.
Imagine AI generates 1,000 browser tests.
If 100 become flaky, the team now has a larger maintenance problem.
Therefore, AI-assisted testing should not focus only on generation.
It should also focus on:
- Reliability
- Failure analysis
- Test maintenance
- Test prioritization
- Root-cause assistance
- Duplicate detection
The system should help keep the automation suite healthy.
---
A Future AI-Assisted Testing Workflow
Imagine:
Requirement changes.
↓
AI identifies affected workflows.
↓
Candidate tests are generated.
↓
High-risk tests are prioritized.
↓
Tests execute.
↓
Failures are analyzed.
↓
AI checks historical patterns.
↓
Potential flaky tests are identified.
↓
Potential product defects are highlighted.
↓
QA engineer reviews the evidence.
↓
Regression suite is updated.
This is a much more complete vision of AI-assisted QA.
---
The Human Still Owns the Decision
Even with advanced AI, a human should be able to review:
What happened?
Why did the system classify it this way?
What evidence supports the conclusion?
What wasn't tested?
Could this be a real intermittent bug?
That transparency matters.
The goal isn't to create a black-box system that says:
"Flaky."
The goal is to create a system that says:
"This test has failed 11 times in 500 executions. Eight failures occurred during API response delays, and the same failures were not associated with application code changes. This may indicate test or environment instability."
That's actionable information.
---
The Real Problem With Flaky Tests
The biggest problem isn't that a test fails.
Tests are supposed to fail when something is wrong.
The problem is when nobody knows what the failure means.
A reliable test creates a clear signal.
A flaky test creates noise.
And when noise becomes frequent enough, teams stop listening.
That's why test reliability deserves the same attention as test coverage.
---
Final Flaky Test Checklist
When an automated test fails, ask:
[ ] Is the failure reproducible?
[ ] Did application code change?
[ ] Did the test code change?
[ ] Did the environment change?
[ ] Is the test data correct?
[ ] Are there timing issues?
[ ] Are asynchronous operations complete?
[ ] Are selectors stable?
[ ] Are tests sharing data?
[ ] Does execution order matter?
[ ] Are external services involved?
[ ] Is the browser relevant?
[ ] Are logs available?
[ ] Is a screenshot available?
[ ] Is video available?
[ ] Is network information available?
[ ] Has this test failed before?
[ ] Does the same failure occur in other environments?
[ ] Could this be an intermittent product defect?
[ ] Should the test be quarantined temporarily?
[ ] Who owns fixing it?
---
Conclusion
Flaky tests are one of the biggest threats to trust in automated software testing.
They create uncertainty.
They waste engineering time.
They slow CI/CD pipelines.
They encourage unnecessary retries.
They can cause teams to ignore real failures.
And over time, they can turn a valuable automation suite into background noise.
The solution isn't simply to delete every test that fails intermittently.
Instead, teams should investigate the underlying cause.
Look for:
- Timing problems
- Race conditions
- Unstable selectors
- Network issues
- Shared test data
- Environment instability
- External dependencies
- Randomness
- Test-order dependencies
- Poor cleanup
- Asynchronous behavior
Then classify the failure correctly.
Is it:
A product defect?
A test defect?
An environment problem?
A data problem?
An external dependency?
Or genuine test flakiness?
That distinction matters.
A strong regression suite isn't one where every test always passes.
It is one where test failures can be trusted.
This becomes even more important as AI-assisted testing becomes more common.
AI can potentially generate thousands of test scenarios.
But if those tests aren't reliable, generating more of them only increases the noise.
The future of AI testing therefore shouldn't be focused only on:
How many tests can AI create?
It should also ask:
How trustworthy are those tests?
AI can potentially help identify patterns in test history, detect repeated intermittent failures, analyze logs, correlate failures with application changes, and suggest possible causes.
But QA engineers still need to validate those conclusions.
The ultimate goal is simple:
When a test fails, the team should want to investigate — not immediately rerun it and hope for green.
That's the difference between having automated tests and having a trustworthy automated testing system.
And as software delivery continues to accelerate, that trust may become one of the most important metrics of all.