Test faster and more efficiently, is it doable?

A software testing strategy based on risks

Software validation for ongoing or corrective maintenance is always risky because of regressions that have immediate impacts.

As the software is already running and used by employees for intern processes, any dysfunction introduced by a new version could quickly become a major issue.

Software regression testing is aiming to avoid these risks and ensure a smooth and risk free transition to the new software release for the company. But the testing area that requires to be considered in order to get rid of regression is often tricky to define. Usually, companies define in their tests management (Quality Center, TestLink, Salome-TMF, Excel, …) a bunch of tests that makes a non-regression tests sequence and they play it for every version.

However, regressions tests involve substantial costs regarding to the global testing effort. Moreover, it does not include changes related risks since the regression test suite always targets the same functional areas.

By having a better visibility on each new version’s content, on the changes that have been done, on the possible regression areas, it becomes possible to optimize the software testing strategy focusing on 2 goals:

• better regressions risks coverage by targeting the efforts on functional domains impacted
• optimizing the test efforts avoiding non relevant tests

Defining a good software testing strategy is a pain

When trying to define the right testing strategy, two major issues are to be fought: the functional part of the software can be huge, and the “reset” of risks for every new version, and eventually every new delivery.

Every delivery can hide regressions

To every delivery is attached a risk of introducing new bugs. That includes the 1st delivery of a new version to test, as well as the last delivery of the validation stage that is supposed to only contain specific patches.

IBM and Caper Jones realized studies that pointed out an 8% of bugs introduced by changes made during the testing stage (related to reported bugs).

These tests imply the knowledge of changes made between two deliveries as well as the potential impacted areas. If functional module A has been changed, which are the impacted modules? Module B? Module C? All of them? Of course knowing the exact impact of a specific change isn’t that simple. The usual testing strategy is then to test all over again. But this requires a lot of time…

This option even becomes impossible at the end of the validation process as there is not enough time to test all over after the last delivery including the latest patches. In this case, it is essential to target the testing efforts on risks with a Risk Based Testing strategy (RBT).

Knowing the risk areas brings factual clues necessaries to define such a strategy and gives the necessary data to to decide to Go/NoGo for the final step.

When regression tests become too much

Along its life, an application tends to grow its functional spectrum and thus its tests needs to eliminate all regressions risks. So the number of scenarios constitutive to the non-regression tests sequence is tending to grow alongside the software in order not to avoid risks.

If it is necessary to redo all the tests for every new version, the testing team is often facing a challenge: play the test sequence in a very short time.

Tests automation seems to be the only possible option to reduce regressions test costs, but most software does not match the criteria making it economically doable and efficient.

Then, without knowing the most critical software areas after a change, it’s frequent to pass a first testing stage with no bugs and then, when tests are almost over, to finally face a highly critical area of the software. So it is a very heavy working load at the very last minute. No fun…

How to test more efficiently

Imagine the testing strategy you would embrace if you knew exactly which parts of the software have been changed, which functional module has been impacted, where the regression risks are…
This is just like knowing what to test in the right order, avoiding surprises at the end of the test process. Better than test automation, it becomes possible to prioritize tests, skip non relevant tests targeting unmodified functionalities.

This new approach of the software is giving the opportunity not to start from scratch for every new test stage, but to rely on tests previously realized. Not every passed test has to be challenged by a new delivery whose exact changes aren’t known. It becomes possible to test better, to “test right” while reducing test efforts.

The Kalistick platform allows you to follow up the evolution of an application. Thanks to that follow up, you can increase tests efficiency while reducing regression risks!