We like to share impressions with our clients. Regarding our platform or even other issues they are facing and possible solutions.
Based on these observations, one thing stands out: improving test efficiency is a major challenge for almost all of them.
One more thing they have in common: trying to achieve better efficiency, they are always slowed down by the same problems, again and again…
We decided to share with you these problems that are poisoning a lot of test teams. Then, we will introduce the solutions that our clients have implemented for their testing.
Why test teams need more efficiency?
Top 3 major and recurring issues
According to our clients, 95% of them encounter 3 major issues while testing:
- Increasing complexity of software: no more monolithic and autonomous applications, welcome to endless inter-integrations (through web services for instance). These integrations are increasing risks with every evolution and make testing more complex. Better testing become unavoidable: despite increasing risks, time and people allocated to software testing remain the same.
- Acceleration of the pace of versions: application’s evolutions are a must for business teams. New versions may contain fewer evolutions, but tests aren’t much easier. Of course there is no way to re-test all over every time. Better test strategies are then essential.
- Endless modification on every level: new functionalities are added to applications; new applications are added to IS… In summary: a ton of new risks are added. Quality requirements might not change, but means are never increased either!
Two weeks ago, a client was telling us about his rather unpleasant experience:
« Every time we go in production, feedback is very fast. We rapidly reach a hundred users, and the way they use the software, if there are hidden bugs, they find them very quick. If the bug is a stop, we got a quick and violent feedback. Covering this risk is a top priority and we must increase our testing efficiency. »
Identified inefficiency sources
Beyond the increasing need of efficiency, our clients share three important sources of efficiency loss during testing:
- A bad visibility on delivered version content
- Low quality modifications
- A high number of iterations to validate a new version
3 cannonballs testers are chained to
No good test without good visibility
In most cases, test teams complain about the lack of visibility on developments made by project teams. The impacts can’t be clearly identified and it is not possible to turn the test strategy toward these risks (or with a very limited confidence).
This bad visibility is due to multiple situations:
- Test teams and dev teams work in a “silo” mode. This mainly happens when development is externalized to a contractor, but not only.
- Developers and testers work together in the same open space; but they don’t speak the same language! When developers say file, class or code, testers want to hear about test case, business need and functional scenario.
The first attempts to improve it are the release notes. Here again, the content is too technical to be used by testers, sometimes too general (just a list of requirements associated to the delivery) or even just declarative and judged not reliable enough.
Consequently, functional validation manager can’t get a clear picture of modifications and their impacts to select the appropriate scenarios to cover risks.
To better understand the consequence of the lack of visibility, here is a customer case:
4 days before validation ending –a last version of the software just arrived. It contains 10 fixes that have to be validated and we have to make sure no regression have been introduced.
D-DAY-4 – evaluation of changes perimeter:
- Are there only the 10 announced fixes? Not 100% sure…
- What are the impacts of these changes on the overall functionalities? We don’t even know the exact changes, so knowing the impacts…
- What about regression risks? Impossible to evaluate accurately. We chose to validate the fixes first, and then we’ll go on the usual priority regression tests.
D-DAY-2 – Regressions are showing up on theatrically unmodified features.
- An urgent new version including new fixes is required!
- We urgently validate these fixes but we can’t go further with non-regression testing.
All good, we go live to production, but… first feedbacks: bugs and unhappy users! Where do the bugs come from? That’s a good question…
Not only this circumstance generates bugs that will be found when the software is on production, but moreover, finding the bug’s source will be particularly difficult.
Low development quality: an obstacle for testers
Development low quality has a strong impact on business test teams. The link might not be obvious, but we often encounter this context:
- Frequent regressions on every delivery as the code is hardly maintainable: any change is very hazardous. A real case with got with many clients: copy / paste in the code. This means more iteration to correct the same bug everywhere!
- Technical anomaly found by business testers that are very hard to reproduce, to document and thus to correct. These anomalies are generally classified as “not reproduced” but surface in production context.
- A development team that prefer delivering on time even if they didn’t pass all the necessary technical tests (sometimes with little unfinished development). As a result, many changes in the next delivery (back to “Lack of visibility point”!) and test results very questionable. A very common regression source!
The lack of quality is not noticeable only in production. With “black box” tests, linking certain issues and their origin is very difficult. Yet, if this “black box” could be opened, the link is obvious!
Number of deliveries: a key performance indicator ?
To every new delivery in a validation stage, there is an associated risk that generally is badly handled. Any change can question the result of an already passed test while the risk won’t even be perceived!
Many studies, like the ones of IBM and Caper Jones, are pointing that the impact of such circumstance is huge:
“5 to 8% of anomalies found in production are coming from changes done during test stage.”
IBM & Caper Jones
In order to avoid this situation, some companies choose to delay the non-regression test campaigns. But it is impossible to delay it forever, and issues always catch them up.
Thus, tests perception is degraded: they look useless whereas test team is doing its best to avoid the problems. The team tries to adapt test plans to each delivery and increase the test efforts when approaching the deadline.
Improving test quality: a dead end ?
There are solutions out on the market that allow test teams to focus and anticipate these problems that undercut their efficiency. We will analyze the solutions our clients implemented in an article series to come. We will talk about how to:
- Get a good visibility on received releases
- Exploit test coverage in order to manage and anticipate risks
- Define a non-regression test strategy really efficient
Meanwhile, if your company is facing problems we didn’t talk about; please share with us in the comments!
