AI Regression Testing: How to Decide Which Tests to Run After a Code Change
A developer changes one part of an application.
The pull request is ready.
Now comes the familiar question for QA:
Which tests should we run?
The traditional answer is often simple:
Run the regression suite.
If the application has 100 tests, this may be easy.
If it has 1,000 tests, it becomes slower.
If it has 10,000 tests, the problem becomes much more complicated.
And if a large organization has hundreds of thousands of automated tests across multiple applications, running everything after every change may not be practical.
This is where AI-assisted regression testing becomes particularly interesting.
The goal isn't necessarily to replace regression testing.
The goal is to make regression testing more intelligent.
Instead of treating every test as equally relevant, an intelligent system can potentially consider:
- What changed
- Which components were affected
- Which features depend on those components
- Which tests cover those areas
- Which tests have historically found defects
- Which tests are reliable
- Which workflows are business critical
- Which tests provide the fastest feedback
The result could be a more focused regression strategy.
But there is an important principle:
AI test selection should not simply mean running fewer tests.
It should mean running the most relevant tests first, while maintaining appropriate confidence in the software.
That distinction matters.
Because the goal of regression testing isn't to execute the maximum number of tests.
The goal is to identify important regressions as efficiently as possible.
---
What Is Regression Testing?
Regression testing is the process of verifying that changes to software haven't unintentionally broken existing functionality.
A change may appear small.
For example:
A developer changes the login service.
But login may be connected to:
- Customer profiles
- Orders
- Checkout
- Account settings
- Permissions
- Subscription management
- Payment workflows
The change may therefore have a much larger impact than the modified code initially suggests.
This is one reason regression testing can become expensive.
The application is interconnected.
A change in one component can affect many other behaviors.
---
The Traditional Regression Testing Problem
Imagine an application with:
5,000 automated tests.
A developer makes a small change to the customer profile page.
The team could run all 5,000 tests.
That gives broad coverage.
But it also creates:
- Longer execution time
- More infrastructure usage
- More test failures to investigate
- More opportunities for flaky tests
- Slower developer feedback
The alternative is to manually select tests.
A QA engineer might think:
"This change affects the profile page, so I'll run these 100 tests."
That can be effective.
But it depends heavily on human knowledge.
The engineer needs to understand:
- Application architecture
- Dependencies
- Previous defects
- Existing test coverage
- Business-critical workflows
As applications grow, manually maintaining this mental model becomes difficult.
---
The Basic Idea Behind Intelligent Test Selection
The basic concept is straightforward.
Instead of:
Code change → Run everything
we can think about:
Code change → Understand impact → Select relevant tests → Prioritize execution
The challenge is determining relevance.
Suppose a developer changes:
`UserProfileService`
Potentially affected areas could include:
- Profile
- Account settings
- User preferences
- Authentication
- Notifications
But not necessarily:
- Product search
- Inventory
- Payment processing
The testing system should ideally understand those relationships.
This is where AI can potentially help.
---
AI Regression Testing Is Not Simply "Run Less"
This is an important distinction.
Some teams may hear:
"AI can reduce the number of regression tests."
and immediately think:
"We're going to reduce coverage."
That's not necessarily the goal.
A better approach is:
Run the highest-value tests first.
For example:
A full regression suite contains 5,000 tests.
An AI-assisted system might identify:
300 directly relevant tests.
Then prioritize:
50 high-risk tests.
Those 50 can execute immediately.
The remaining relevant tests can continue afterward.
This creates faster feedback without automatically abandoning broader coverage.
---
The First Signal: What Changed?
The first question an intelligent regression system should ask is:
What changed?
This may involve:
- Source-code changes
- API changes
- Database changes
- Configuration changes
- Dependency updates
- UI changes
- Infrastructure changes
A small code change can have a large impact.
A large code change can sometimes have a limited impact.
Therefore, simply measuring the number of changed lines isn't enough.
Context matters.
---
Code Change Impact Analysis
Suppose a developer changes a function responsible for calculating discounts.
At first glance, the change may be limited to one service.
But the output could affect:
- Product pages
- Cart totals
- Checkout
- Order confirmation
- Invoices
- Customer emails
A good regression strategy should understand those relationships.
This is sometimes called change impact analysis.
AI can potentially assist by combining code structure with historical testing information.
For example:
Changes to this service have historically affected these workflows.
That provides useful context.
---
Static Analysis Has Value
One possible source of information is static analysis.
The system can inspect relationships between:
- Files
- Functions
- Classes
- Modules
- Services
- APIs
This can help identify what might be affected by a code change.
For example:
`PaymentCalculator`
↓
`CheckoutService`
↓
`OrderService`
↓
`OrderConfirmation`
A change to `PaymentCalculator` may therefore suggest testing:
- Cart totals
- Checkout
- Order creation
- Confirmation
Static relationships provide a structural view of potential impact.
But they aren't enough by themselves.
---
Historical Testing Data Adds Another Layer
Now imagine the system also has historical information.
It knows:
Whenever `PaymentCalculator` changed in the past, these tests frequently detected regressions:
- Checkout total
- Discount calculation
- Tax calculation
- Order confirmation
That's valuable.
Static analysis tells us:
What could be affected.
Historical testing tells us:
What has actually shown risk before.
Combining the two can potentially produce better test selection.
---
Historical Failures Are Useful Signals
Imagine two tests cover the same component.
Test A
Passed 5,000 times.
Never found a defect.
Test B
Passed 4,900 times.
Found three meaningful regressions after similar code changes.
If the component changes again, Test B may deserve higher priority.
This doesn't mean Test A is useless.
It means historical outcomes can provide useful context.
AI can potentially analyze those patterns at a scale that would be difficult to maintain manually.
---
Business Risk Should Also Matter
Technical dependency isn't the only consideration.
Imagine a code change affects:
Customer profile image upload
and:
Payment authorization
Both may have similar technical change impact.
But their business risk isn't equal.
Payment authorization may have:
- Revenue impact
- Customer impact
- Security implications
- Regulatory implications
Therefore, intelligent test prioritization should ideally consider business risk.
Potential signals include:
- Revenue
- User volume
- Criticality
- Security
- Regulatory requirements
- Historical incidents
- Customer impact
This is where QA strategy remains extremely important.
---
Test Reliability Should Influence Selection
Now add another signal:
Test reliability.
Suppose a test has a 10% intermittent failure rate.
Another test has a 99.9% reliable execution history.
If both are selected for regression, their results shouldn't necessarily be interpreted equally.
The unreliable test may generate noise.
The reliable test may provide stronger evidence.
This is why test selection and test reliability are connected.
An intelligent regression system should understand not just:
Which test covers this code?
but also:
Can I trust this test's result?
---
Test Selection Should Consider Test Value
Not all tests provide equal information.
Consider two tests.
Test A
Checks whether a page title is correct.
Test B
Verifies that a customer can successfully complete a payment.
Both are tests.
But their business value is very different.
If execution capacity is limited, high-risk workflows should usually receive greater attention.
This is why test prioritization should consider:
Coverage + risk + reliability + historical value.
---
Test Execution Time Matters Too
Imagine:
Test A takes 10 seconds.
Test B takes 15 minutes.
Both cover relevant behavior.
If the goal is fast feedback, Test A may be useful to run immediately.
Test B can run afterward.
This suggests another useful metric:
Risk-adjusted feedback speed.
The best first tests aren't necessarily the most comprehensive.
They may be the tests that provide the highest-value information quickly.
---
Regression Testing Can Become a Queue
Imagine a pull request arrives.
Instead of treating the regression suite as one large block, an intelligent system could organize it into priority levels.
Priority 1
Critical and directly affected tests.
Run immediately.
Priority 2
Highly relevant related tests.
Run next.
Priority 3
Historically useful tests.
Run afterward.
Priority 4
Broad regression coverage.
Run continuously or before release.
This approach doesn't eliminate the complete suite.
It changes the order in which information arrives.
---
Faster Feedback Is Often the Real Goal
Developers don't necessarily need every test result before making every decision.
They need useful information quickly.
Imagine:
Code change at 10:00.
At 10:05:
20 high-risk tests completed.
One critical failure detected.
The developer can start investigating immediately.
Without intelligent prioritization:
Wait 90 minutes.
Then discover the same failure.
The difference is feedback time.
That's where intelligent regression testing can create significant value.
---
AI Can Potentially Learn From Previous Releases
Historical releases contain valuable information.
Suppose a team has 100 releases.
AI could potentially analyze:
- What changed
- Which tests ran
- Which tests failed
- Which defects were discovered
- Which tests caught them
- Which areas were affected
Now the system can identify patterns.
For example:
Authentication changes have historically affected checkout and customer-account workflows.
Or:
Changes to pricing logic frequently cause failures in regional tax calculations.
This historical context can improve future test prioritization.
---
Production Defects Can Become Testing Signals
Production incidents are some of the strongest evidence available.
Suppose three production incidents involved:
Coupon calculation
That information should influence future testing.
When coupon-related code changes again, the system should know:
This area has demonstrated risk.
AI can potentially connect:
Production defects
↓
Code areas
↓
Features
↓
Tests
↓
Future changes
That creates a feedback loop between production quality and pre-release testing.
---
The Feedback Loop
A mature regression system could eventually work like:
Code change
↓
Impact analysis
↓
Test selection
↓
Risk prioritization
↓
Execution
↓
Failure analysis
↓
Defect confirmation
↓
Production feedback
↓
Historical learning
↓
Better future prioritization
This is much more powerful than simply running the same test suite after every change.
---
What About Tests That Have Never Failed?
Should a test that has never found a defect be removed?
Not necessarily.
Absence of historical failure doesn't mean absence of value.
A test might protect a critical business rule that simply hasn't been broken recently.
For example:
Payment authorization may have zero failures for a year.
That doesn't make the test unnecessary.
Risk still matters.
Historical failure data should be treated as one signal, not the only signal.
---
AI Shouldn't Become Overconfident
This is another important principle.
Suppose AI says:
"These 20 tests are sufficient."
That is a dangerous statement without context.
A better statement would be:
"These 20 tests are the highest-priority tests based on change impact, historical failures, business criticality and test reliability. The remaining regression suite is still recommended before release."
That is much more responsible.
AI should assist with prioritization.
It shouldn't create false certainty.
---
Regression Selection Should Be Explainable
If an AI system chooses 50 tests out of 5,000, engineers should be able to ask:
Why these 50?
The system might explain:
- 15 directly cover changed components.
- 10 cover dependent services.
- 8 have historically caught similar regressions.
- 7 represent critical business workflows.
- 6 have high historical defect detection value.
- 4 are high-priority security scenarios.
Now the QA engineer can review the logic.
Explainability is important because test selection affects release confidence.
---
AI Test Selection Should Support Human Overrides
There will always be situations where human knowledge is better.
For example:
A QA engineer knows:
"We're running a major marketing campaign tomorrow, so checkout deserves additional testing even though the code change doesn't directly affect it."
AI may not know that.
Therefore, a mature system should allow:
- Manual test inclusion
- Manual exclusion
- Risk overrides
- Priority adjustments
- Business-event context
AI should make decisions easier.
It shouldn't make humans powerless to override them.
---
The Difference Between Test Selection and Test Prioritization
These concepts are related but not identical.
Test selection
Which tests should run?
Test prioritization
Which tests should run first?
You can prioritize the entire regression suite without necessarily removing anything.
For example:
5,000 tests
↓
Top 100
↓
Next 500
↓
Remaining 4,400
This may be a safer initial approach than permanently excluding tests.
---
Start With Prioritization Before Reduction
For teams worried about coverage, this is a useful adoption strategy.
Don't begin with:
"AI will remove 80% of our regression tests."
Begin with:
"AI will tell us which tests should run first."
That's much easier to trust.
If the first 100 tests consistently identify the most important problems, confidence grows.
Then teams can evaluate whether some tests can be run less frequently.
---
AI Regression Testing Needs Good Test Metadata
For intelligent prioritization to work, tests need useful metadata.
For example:
- Feature
- Component
- Business risk
- User journey
- Browser
- Environment
- Execution time
- Historical reliability
- Defect history
Without this context, AI has less information to work with.
This means organizations may need to improve their test architecture before AI can deliver maximum value.
---
Test Intent Can Become Metadata
A test should ideally communicate what it protects.
For example:
Verify that authenticated customers cannot access another customer's order.
This is much more useful than:
Test 4921.
If AI understands test intent, it can potentially reason about relevance.
A change to authorization logic should immediately suggest security-sensitive tests.
This makes semantic test descriptions increasingly valuable.
---
AI Regression Testing Should Work Across Layers
A modern application may have:
- Unit tests
- Integration tests
- API tests
- UI tests
- Visual tests
- End-to-end tests
A change may affect several layers.
For example:
A backend pricing change may require:
Unit tests
↓
API tests
↓
Checkout integration tests
↓
End-to-end checkout tests
The best regression strategy shouldn't automatically prioritize only browser tests.
It should understand the testing pyramid and the purpose of each layer.
---
Unit Tests Can Provide Fast Feedback
If a change affects a calculation function, relevant unit tests can execute quickly.
That's valuable.
You don't necessarily need to wait for a browser workflow.
A good regression strategy may therefore prioritize:
1. Fast directly affected tests
2. Relevant integration tests
3. Critical API tests
4. High-risk end-to-end tests
5. Broader regression
This can dramatically improve feedback speed.
---
End-to-End Tests Provide Different Evidence
End-to-end tests are expensive but valuable.
They verify that multiple components work together.
Therefore, an intelligent regression strategy should not treat them as interchangeable with unit tests.
Instead, understand:
What information does each test provide?
This brings us back to the central idea:
Testing is about evidence.
---
AI Could Help Optimize the Testing Portfolio
Over time, an organization may discover:
- Some tests are redundant
- Some tests are unreliable
- Some tests are extremely valuable
- Some tests are expensive
- Some tests are rarely relevant
- Some areas lack coverage
AI can potentially help analyze the entire portfolio.
Instead of asking:
"How many tests do we have?"
ask:
"What information does our test suite provide?"
That's a much more useful question.
---
Regression Testing Is Also a Cost Optimization Problem
Suppose running the full regression suite costs:
- 3 hours of infrastructure
- 2 hours of engineer waiting time
- Significant CI resources
If an intelligent strategy can deliver meaningful feedback within 20 minutes, the organization may gain substantial productivity.
But cost reduction should never come at the expense of critical coverage.
That's why risk needs to remain central.
---
A Simple Risk-Based Scoring Model
An organization could start with a basic conceptual score.
For each test, consider:
Change impact
How closely is the test related to the changed code?
Business risk
How important is the behavior?
Historical defect value
Has the test found meaningful problems before?
Reliability
How trustworthy is the test?
Execution cost
How long does it take?
A conceptual priority score could combine these signals.
For example:
Priority = impact + business risk + historical value + reliability − execution cost
The exact mathematics will vary.
The important idea is that test priority should be based on multiple signals rather than one metric.
---
Don't Overcomplicate the First Version
Teams don't need a sophisticated AI model on day one.
A practical system could begin with:
- Changed files
- Test-to-component mapping
- Business-critical tags
- Historical failures
- Test reliability
- Execution time
That's already useful.
AI can then help interpret and combine these signals.
Over time, more context can be added.
---
A Practical AI Regression Workflow
Here's a simple model QA teams can use.
Step 1 — Detect the change
Identify files, services, APIs or features affected.
Step 2 — Map dependencies
Find components and workflows connected to the change.
Step 3 — Map tests
Identify tests covering those components.
Step 4 — Add business risk
Prioritize critical workflows.
Step 5 — Add historical evidence
Look at previous failures and defects.
Step 6 — Add test reliability
Reduce confidence in unstable tests.
Step 7 — Add execution cost
Prefer fast high-value tests for early feedback.
Step 8 — Prioritize
Create execution groups.
Step 9 — Execute
Run the highest-priority tests first.
Step 10 — Analyze
Investigate failures with evidence.
Step 11 — Expand
Run broader regression as needed.
This is a much more intelligent approach than simply:
"Run everything."
---
What AI Should Tell the QA Engineer
Imagine a pull request modifies the checkout service.
The system could provide:
Change impact
High
Directly affected workflows
Checkout, payment, order creation
Recommended immediate tests
42
High-risk tests
18
Historically valuable tests
12
Potentially flaky tests
5
Estimated priority-run time
18 minutes
Broader regression
Recommended after priority suite
This turns regression from a large undifferentiated test collection into a decision-support system.
---
AI Doesn't Remove the Need for Full Regression
This point deserves emphasis.
There will still be situations where full regression is appropriate.
For example:
- Major releases
- Large architectural changes
- Database migrations
- Authentication changes
- Payment changes
- Infrastructure changes
- Major frontend redesigns
Intelligent selection doesn't mean:
Never run everything.
It means:
Don't assume everything must always run immediately.
---
The Best System Is Adaptive
Testing strategy should change depending on the situation.
Small low-risk change
Run focused tests.
Medium feature change
Run affected tests + related regression.
High-risk change
Run broad regression.
Major release
Run comprehensive validation.
AI can potentially help classify the change and recommend the appropriate level.
But humans should retain the ability to override the recommendation.
---
Why This Matters for CI/CD
Fast delivery requires fast feedback.
If every pull request waits for a massive regression suite, development velocity suffers.
But if testing is reduced too aggressively, defects escape.
Intelligent test prioritization sits between those extremes.
The objective is:
Fast feedback without reckless reduction in coverage.
That's the real challenge.
---
The Future of Regression Testing
I think regression testing is moving toward a more dynamic model.
Instead of:
Static suite → Execute everything
we may increasingly see:
Change → Analyze impact → Understand risk → Select tests → Prioritize → Execute → Learn
The test suite becomes less like a fixed checklist.
It becomes a dynamic system.
---
AI Could Make Regression More Context-Aware
Imagine the system knows:
A developer changed payment logic.
A major sales campaign starts tomorrow.
The payment service has experienced three incidents this month.
Checkout is responsible for significant revenue.
The relevant tests have historically caught regressions.
Now the system has a much richer understanding of risk.
This is far more useful than simply knowing:
"20 files changed."
---
The Human QA Engineer Still Matters
AI can analyze enormous amounts of data.
But humans understand things that may not exist in the data.
For example:
- Upcoming business events
- Customer complaints
- Product strategy
- Regulatory concerns
- Known organizational risks
- New features not yet represented historically
Therefore, the best model is collaborative.
AI recommends.
QA reviews.
The system executes.
AI analyzes.
Humans decide.
---
The Goal Is Not Fewer Tests
This is worth repeating.
The goal of AI regression testing isn't:
Reduce the number of tests.
The goal is:
Reduce unnecessary waiting while preserving useful confidence.
Sometimes that means running fewer tests.
Sometimes it means running the same number in a smarter order.
Sometimes it means running more tests because the change is high risk.
The strategy should adapt to the situation.
---
A Better Definition of Intelligent Regression Testing
I would define it like this:
Intelligent regression testing is the use of application changes, dependencies, historical evidence, business risk and test reliability to determine which tests should provide feedback first and which broader tests are appropriate afterward.
AI can help make this process scalable.
But the underlying principle isn't new.
It's good risk-based testing.
AI simply gives teams the potential to apply it at much greater scale.
---
Final Thoughts
Regression testing isn't going away.
If anything, software development is becoming faster and more complex, which makes regression confidence even more important.
The challenge is that test suites are becoming larger.
Running everything after every change can become expensive.
Running too little can allow defects to escape.
That's the balance AI-assisted regression testing is trying to improve.
The most useful systems won't simply say:
"Run fewer tests."
They'll say:
"Here's what changed."
"Here's what could be affected."
"Here's what has historically been risky."
"Here's which tests are reliable."
"Here are the highest-priority tests."
"Here's what failed."
"Here's what the evidence suggests."
And:
"Here's whether broader regression is still necessary."
That's a much more powerful model.
The future of regression testing may therefore move away from the idea of a fixed list that runs in the same way after every change.
Instead, regression can become dynamic.
The suite responds to the software.
The priority responds to risk.
The evidence responds to failures.
And the strategy responds to what the team is actually trying to protect.
AI can potentially make that possible at a scale that would be difficult for humans to manage manually.
But the goal should never be to let AI decide that fewer tests are automatically better.
The goal is to make every test decision more informed.
Because ultimately, regression testing isn't about proving that every test passed.
It's about gaining enough trustworthy evidence to answer a much more important question:
"Can we confidently release this change?"
And that's where AI-assisted testing has the potential to become much more than automated test generation.
It can become a layer of intelligence between code changes and release decisions.
That is the direction worth exploring.