To assess the maturity of an organization or a team on testing is often a starting point for our customers to implement improvement plans. The implementation and monitoring of these plans rely then on indicators, to ensure progress and results.

Our test expert partners are frequently asked to achieve these audits, develop plans, and implement indicators to follow. We worked to build dashboards that can contribute to this audit phase but also be a support for the monitoring of actions and showing the results.

These charts are based on the information generated for each project. They provide several ways to show this information with indicators:

• test coverage on overall application or only the modifications
• the number of iterations/delivery needed for validation
• the distribution of the testing effort between different types of tests, whether manual or automated, performed by the development team or the functional testers
• and other indocators as timing, modification rate or coverage evolution…

Portfolio Cartography

The Portfolio is available on the cockpit via the “Dashboard” menu, Mapping is available in the “ Evolution ” part of the Portfolio (see left menu).

A major way to show the information is based on a 2-dimensional graph. Its goal is to compare the testing activities of several projects, to highlight risky ones and to identify improvement areas.

Each quadrant has a specific meaning and you can choose what kind of information you want to display: information about the project or about the latest target version. The size of each disk is the size of the project or the latest version if you want to focus on the changes scope. The color represents the test coverage either on the overall application or only modifications.

Quadrant 1 « Weakly tested »

This area includes projects with a lack of test. Not only there is little testing in the development phase but also bit of functional tests. If further analysis confirms a quality problem, several improvement actions can be taken in the short and medium term.

Quadrant 2 « Especially unit and integration tests»

In this area, there are projects with development teams having reached a good maturity on testing. It usually means testing in a continuous integration process. There are more often agile teams, they choose to automate their tests and to do less manual functional test. There are also applications that mainly offer services (SOA) and therefore does not provide a user interface. However, there is little functional tests with end-user vision or end to end vision, so we can expect some bad surprises after the production delivery.

Quadrant 3 « Especially functional tests »

Here, the projects are properly tested but certainly mainly by functional tests or end-to-end tests. And it is likely that there are mainly manual tests. Applications quality should be fine in production.
However there are 3 disadvantages :

• These are functional tests executed at the end of the development cycle. As there is little testing during development, problems arise at the end of the cycle and tend schedules. This increases the risk that a future version goes into production being not totally stable, e.g. if we do not have time to run all these tests.
• The cost of quality for these applications is important because correcting a bug during functional testing is 10 to 100 times more expansive than correcting it upstream.
• Furthermore, in the future it will be difficult to automate these tests, because they will probably be change sensitive and therefore generate high maintenance costs.

Quadrant 4 « Good security »

Application located in this quadrant certainly has a satisfactorily high quality. The cost of quality must be reduced. Unit and integration tests ensure quality throughout the implementation and functional tests assure that the complete business process works properly.

Different interpretations of the presented data can support more comprehensive analysis and identify specific improvement plans.

See web documentation about Portfolio.

When it comes to make evolutions or patches on an application, several questions arise frequently:

• What are the burdens associated with development and testing to confirm the budget and schedule?
• What are the risks associated with these changes? Because, if there should be a huge testing effort, it can be preferred to postpone these changes to a more important release in which this effort will be easier to absorb.
• What is the impact on the work of the testing team? Is there a lack of test already affecting these functionalities? Should you anticipate the maintenance of automated tests related to these changes?

The people in charge of answering these questions often lack factual analysis support for their analysis. And those who have to validate these answers appreciate precise elements to support their decision. This need was expressed several times by our customers whether between a subcontractor and his client or between the contracting authority and the prime contractor, but also by an editor between head management and R&D management. Customers, before us, have identified the interest of data collected by our platform to meet this need. So we responded with a new feature that we called “Impact Simulator”.

The Impact Simulator is used to evaluate, from an area to change (features, requirements, code subset) the impacts related to changes in terms of development, testing and risks.

Development costs estimation

Estimate the burden of developing a new feature is based on analysis of the application and its implementation. However it is not always easy to make reliable estimates or support these estimates with precise quantifications. On the other hand, we need to compare these estimates to identify risks that may be mispriced or otherwise too generously evaluated.

Our solution analyzes all changes made to each version and the elements involved. Thus it provides an analysis of what has actually been done with the given budget. The Impact Simulator reverses the situation and assesses, from a given perimeter to be changed (features, requirements, code subset), the impacts related to the changes. We don’t give miracle answers but reliable indicators to refine the estimate or compare with other estimates or previous versions.

Test cost estimation

Usually the test burden is evaluated using different methods:

• As a ratio of development costs, 10, 20, 50% or 100% depending on project, risks, etc.
• As an incompressible cost, e.g. performing a complete regression campaign, 3, 8, 20 days or even several months on some waterfall projects…
• With estimates based on requirements or risks (RBT as Requirement Based Testing or Risk Based Testing).

Behind these estimates lie different types of testing efforts. It can be the unit tests or integration tests achieved by the development teams. Or it can be the functional tests, manually or automatically designed, maintained and carried out by the contracting authority. However in terms of tests concerned by these developments, the differences are also in the existing:

• Is there any existing manual test and can we predict the total execution time?
• Is there a lack of test? Do we have to foresee the design of new test cases or minimum scheduled time for exploratory testing?
• Is there any automated functional test? Can we predict their possible maintenance? (cf. zoom on a customer case below).

The response of the Impact Simulator is simple: according to the area affected by these developments, it identifies the impacted tests whether manual (to be executed again) or automatic (to be maintained). It also identifies the “test holes” that is to say, the current lack of tests. Thus it becomes easier to anticipate the test burden and its various dimensions.

Identify the risks associated to a release

A classic but frequent use of the impact analysis is to identify the potential side effects of an evolution or a fix development. These may seem simple at first, but they have sometimes an impact greater than expected. The consequences are: heavier development costs, more tests to execute and even laborious production delivery.

To perform a rigorous risk analysis, the Impact Simulator uses two Kalistick platform features:

• The business model of the application that represents the functional subsystems: it will bring up the elements impacted by future changes. Analysis can be done based on requirements or risks when such a model is used with tools like HP Quality Center.
• The footprint of functional tests: the simulator identifies functional tests impacted by future developments. It also calculates the “number of hits” to identify the most affected.

Presentation of the Impact Simulator

The Impact Simulator uses the data captured by the Kalistick platform to meet the needs outlined above. The data consist of different versions of the application’s scan and delivered test footprints performed during these earlier versions. Changes or fixes to be carried out will be specified from the functional model (modules / requirements / risks impacted) or from the results of a technical study indicating the affected code elements (files, packages, classes, methods…).

Define the impacted scope

The impact analysis is based on the identification of the elements affected by the developments. Two solutions are proposed for this:

• Select business modules
  this identification is based on the definition of requirements or features. It uses the business model of the application built in the Kalistick platform. Thus in the example below showing the Workflow module, the platform finds all code elements inside this module and that are in the scope to evolve.

   

 

• Specify code elements
  

  This second option allows you to specify directly the code to evolve. This assumes that the technical team has already made ​​a first technical assessment for providing this list.

Then the analysis must be launched by clicking the « Analyze Impact » button:

Result interpretation

The simulator first displays figures about the evolution in terms of number of affected tests, execution time of these tests, test holes (lack of test) or simply the proportion of changes with respect to the overall size of application.

“Impacted Tests” are the tests whose execution footprint uses elements of code to be modified. The estimated time for the execution of these tests is noticed. This estimate is based on the time of the last execution of each test.

The list of “Impacted Modules” provides a risk assessment. Each module that contains code to be modified is identified; the number of impacts is an indicator of the number of modified elements in every single one of them.

The list of “Impacted Tests” presents the number of impacts and the last execution duration for each test.

The “Impacted Code Elements” are all the elements taken into account for the analysis. For each element the list displays the volume of the element (lines of code), which assesses the risk or the burden.

Finally, the “Testing holes” are all elements of the code concerned by the evolution and for which there is no functional test. Working with the technical team is required to assess the risk associated with this lack of testing and the need for the development of new test cases.

The “Nearby tests” function identifies, for a given element, test case footprints that pass near it in the source code. Thus, these tests may be slightly modified in order to cover the given element. It also gives a first idea of ​​the functional role of this code in the application.

Conclusion

With the Impact Simulator the Kalistick Solution meets an upstream need in agile project as well as in waterfall project. It allows anticipation to make or confirm costs evolution and subsequent decisions based on facts. This new feature is already deployed and accessible to all our customers on their Kalistick Cockpit, and here is the link to access this feature.

While most market performance and testing tools face the extraordinary complexity of the Dassault System Enovia Enterprise Software on which the Faurecia PLM is based, the Group discovers  Kalistick, an innovative solution which helps them greatly streamline their testing process and optimize their software quality.

Feedback on the tool deployment with Jean-Noël Gueutal, PLM  Development Team Manager at Faurecia.

Faurecia

Zoom on the Faurecia case

The Challenges

This project depends on 2 major stakes:

• Streamline the testing process andreduce cost/time all the while improvingthe efficiency of regression testing
• Adopt a new testing tool in a particularly complex context

Kalistick’s Solution

Kalistick, a tool to increase the quality of Product Lifecycle Management

Faurecia’s size and the importance of its market require its Product Lifecycle Management (PLM) to be rigorous and of the highest quality. This is the responsibility of the Engineering Information Technology (EIT) Department, directed by Eric Borgard, and split into two divisions.

The first one deals with the functional aspects, regarding jobs and key users. The second one manages the technical aspects: specifications, development, packaging, and testing, as per a classic waterfall model.

For Jean-Noël Gueutal, PLM Development Team Manager at Faurecia and therefore responsible for the second division, the PLM software quality is a fundamental aspect of equipment design for the automotive market.

“It is in this context that we became interested in Kalistick. For the PLM application, we already perform more than 70,000 automated tests with the HP QTP tool. We also use HP QC (Test Management Solution) to formalize our manual testing.”

The Kalistick solution, which can detect testing gaps between different software versions, quickly turns out to be a particularly interesting tool to build new scenarios on missing tests, and thereby improve test coverage on the Group’s PLM software.

A significant challenge: compatibility with the Dassault Enovia System Entreprise Software

If the teams were initially skeptical, it was also due to the particularly complex and non-standard structure of the Dassault System Enovia Enterprise Software on which the Group’s PLM is based..

One simple fact gives an idea of the size of the Kalistick’s challenge: the last upgrade of the Group’s PLM required about 500 days of development without the addition of any new features.

As a first step, a Proof of Concept was undertaken. “I admit that we were positively surprised to see that Kalistick succeeded the first time round to capture 100% of the testing footprint and to detect the impact of changes,” says Jean-Noël Gueutal. The POC was therefore approved by the Group and the implementation of the solution was confirmed.

Implementation conditions to which Kalistick adapts perfectly

Kalistick’s ability to integrate with the major test automation and test management solutions, HP Quality Center (QC) and HP Quick Test Pro (QTP) respectively, were also a key consideration for the adoption of the solution by Faurecia.

It is easily understood: for a company that generates more than 70,000 tests with these industry-leading solutions, it is essential that any new tool integrate with them perfectly, as well as bring significant and complementary value to the table.

Finally, Kalistick’s hybrid ‘On-premises/Cloud’ architecture is another strong point for the solution: the ‘Application Scan and Test Fingerprint Capture’ modules which are installed at the customer site are simple to put in place, while the analysis and the dashboard are hosted in the Cloud.

“The fact that Kalistick does not require the installation of an internal server is a plus,” Mickaël Bertrand points out.

Kalistick, a key asset to switch to agile mode

But beyond these purely technical considerations, Kalistick could very well be a significant asset in the development planned by the Group’s EIT Department.

“We plan to move from the traditional waterfall method to agile methods,” explains Jean-Noël Gueutal.

“Furthermore, in order to deliver new releases of the PLM more often. We will therefore need to further optimize testing. Not only can Kalistick detect testing gaps downstream to help us build new complementary scenarios, it also allows for a better selection of tests to be run upstream, optimizes them (by avoiding testing the same thing several times) and prioritizes them, which is absolutely critical for short-cycle releases.”

To industrialize its testing process and improve its software delivery quality, SQLI Group chose the Kalistick solution.

Let’s take a look at how the solution was deployed within the company on a major e-commerce project for a large international retail chain, with Franck du Verdier, SQLI Group’s Program Director.

SQLi Group

Zoom on an SQLi Project

• Hybris platform
• Agile Development (Scrum)
• Biweekly delivery of new versions
• Offshore Resources

The Challenges

Kalistick’s Solution

Demand for quality and speed

It was a major challenge. The group has an especially large presence in e-commerce, where demand for quality sites is intense, because the smallest flaw or bug could quickly translate into a heavy loss of sales.

Furthermore, this industry segment requires strong responsiveness and continuous adaptation of applications as marketing departments need to constantly integrate more intelligence and more functionalities into their portals. So for SQLI, Kalistick had to demonstrate  quickly and very concretely its solution’s added value vis-a-vis these challenges.

Software Testing Industrialization at SQLI

Very quickly, the advantages of the solution became sharply apparent to SQLI. By combining the various layers of unit tests, integration tests, and systems tests implemented during the agile development cycle, Kalistick allowed them to  identify very clearly the “testing gaps” and therefore the potential quality gaps between different versions of the site. But it goes further.

Because each test’s exact footprint is captured in the code, the solution is able to tell which tests must be  run and which ones are useless because they are not related to any code modifications and not affected. Since e-commerce applications evolve very rapidly, especially at the user interface level, the tests are difficult to automate, which means  there are still many manual tests.

“This saves us a significant amount of time!” Mr. du Verdier adds. “Whereas we used to have to run all the tests, now we concentrate on the ones that matter, and we reduce the delivery time for each version.”

And it goes even further than that:

How to reconcile quality improvement, cost control and agile methods

For a group like SQLI, which has been evolving toward Scrum-type agile methods for a long time, this is truly a fundamental point.

Development cycles are very short, and new versions are delivered every two weeks. But the thing that is  greatly valued in the hyper-responsive e-commerce environment is a major constraint in terms of quality. It is simply impossible to execute a complete set of tests every two weeks without increasing human resources costs. Kalistick is a real  solution at this level.

Improve offshore resources monitoring

But according to SQLI the advantages of the solution don’t stop there.

“Thanks to the test coverage intelligence that the tool provides, we have a better quality indicator for the versions we produce from sprint to sprint,” Mr. du Verdier explains. “We can maintain strict monitoring, even over our offshore resources.”

Kalistick is a solution that makes it possible to move from quality indicators that  quantitatively measure average testing effort (time spent, number of tests carried out, number of anomalies detected…) to indicators that measure results in terms of quality control, such as coverage  of modifications made to a version.

“We are alerted before delivery if we need to reinforce a testing effort,” he adds. “Without Kalistick, the testing gaps would not always be properly detected, and we  would have to wait for the client to report bugs in order to tailor our efforts.”

Bringing more transparency to the client

In a more general way, Kalistick is also perceived as an added value worth mentioning to the final client.

Kalistick brings attractive guarantees of quality and transparency to clients who use  Third-Party Application Management Services.

“In this case, an e-commerce site created by SQLI Group,” says Mr. du Verdier, “we did not hesitate to let our client know that we were using Kalistick,  and they really appreciated this choice for their project.”

Janet Gregory

The goal is to ensure application quality by detecting most regressions while running fewer tests. It seems impossible to win simultaneously on quality & time? Let’s see how impact analysis can increase testing effectiveness by 30%.

Regression Testing

It is often a headache. Regression testing represents an important part of the overall test effort on each application release. They are required because they detect problems that have been undetected upstream. However they also often miss bugs which are found after deployment when the cost to fix them is usually estimated to be at least 10 times higher (diagnostic, fix, test, deploy, business impact, etc.).

Regression testing effectiveness is required to:

- Reduce the risk of bugs in each release and thus select the right tests to execute in the time frame dedicated to regression testing.

- Be responsive to business needs with faster release and smaller release. This requires also faster regression testing and to be able to design effective regression test campaigns according to the content of each version.

- Adapt to financial constraints

How much regression testing?

For most projects, even small ones, the number of tests required to ensure that there are no regressions can quickly become unmanageable. The following diagram illustrates this situation if all tests created for a specific are always added to the regression test suite.

How to define a regression test strategy?

It is recommended to include a specific approach for regression testing in the test strategy in order to limit the number of tests, usually it includes mainly two types of tests:

1. Tests related to major features of the product with a priority regarding requirements and risks.

2. Tests related to features that have been proven to be less robust (where bugs and/ or regressions were found in previous releases).

However, if these criteria reduce the number of tests, they do not adapt the strategy to each software release. Thus it leads to execute always the same test suites and the result is often that regressions are not detected. How to improve?

Test automation.

Automated tests appear to be the Holy Grail: execute all tests, on every release within a limited timeframe and for a fraction of the costs. A greater number of tests should lead to a better functional coverage and thus a better quality.

Reality is quite different because statistics made on our platform show that 80% of projects do not have any automated functional tests, 5% have intensive automation strategy, and 15% focused and limited test automation.

Why so few automated tests?  Customers explain that it’s not always faster and cheaper because test maintenance is a significant burden.  Test automation could be the right choice but it has to be done with a well structured approach.

Impact analysis to optimize software testing

Why impact analysis?

When a new release of an application is done, whether corrections, changes, or new features, development teams are making changes to the application code. It is these changes that generate regressions on existing features.

Links between changes and tests?

Our research done in collaboration with several research centers shows that the probability to find a bug with a specific test is directly related to the volume of code modification that this test will use when you execute it.

The impact analysis goal is therefore to establish links between changes and tests. This analysis is impossible without a tool, however, existing tools do some technical analysis but they don’t provide any result at the test level.

To meet this need, Kalistick have developed “Test Scoring”. At each release of the application, each test is scored according to the volume of changes that affects its execution. This score is based on three unique techniques:

- Capturing Test Footprints. Whenever a tester executes a test, Kalistick’s system automatically captures all code executed within the application for this test. This footprint gives all the links between the test and the code; it is stored in a knowledge base.

- Change detection.  Kalistick scanner analyzes each release and detects changes by comparing the scan results with previous ones.

- Kalistick engine analyzes Test Footprints to assess impact of changes and compute Test Scores.

The principle is the following: as soon as a test is executed the system stores it. At each future version it is scored to assess whether it is required to execute it.

Customers use the scoring within their usual testing tools like HP Quality Center, HP ALM, Test Link, XStudio, ReferTest, etc. Kalistick’s plug-ins provide seamless integration, so test teams can rely on it to boost the effectiveness of their test campaigns.

Results

It can increase test effectiveness over 30%. This is measured on two axes: gain in quality and time saving. Two customers cases to illustrate these benefits:

- A client executes  its usual regression test suite and then it selects 10 tests to be executed using Kalistick scoring. Result: 3 regressions detected within 3 hours.

- Another client measures the quality of each release according to the number of bugs detected in production within 3 months after deployment. Result on the first release tested using Kalistick, the number of bugs has been down by 50% leading to saving of tens of thousands dollars.

To go further, you can download the white paper “30, a key number for test effectiveness ” and the product sheet “Intelligence Test Coverage” which highlights key features.

And get the product sheet for more details.

You know the famous Swiss cheese

If you are not a cheese specialist, Swiss cheese is a generic name for cheese with holes like you can see on the following picture which is a slice of Swiss Emmental.

Testing layers from development to deployment

To introduce you to the Testing Swiss Cheese Syndrome, let’s think about all the testing activities going on during an application development. It is usually a subset of unit testing, integration testing, functional testing, automated testing, manual testing, exploratory testing, etc.

It is very unlikely that a single kind of test can cover the whole application. That’s where I see the similarity with a slice of Swiss cheese: if you look at your test coverage you will find gaps (like holes), areas that haven’t been covered by any test.

From development to deployment, testing activities are done by different people like developers for unit tests or QA testers for functional tests. They use different tools adapted to their own testing practices like unit tests frameworks (JUnit, …), web services testing solution (SoapUI, …), functional test automation tools (HP QC, Test Complete, Selenium, …), Test Management Solution (TestLink, HP QC,…).

All these testing layers are added on the top of each other, and in each slice you will find some testing gaps.

The Swiss Cheese Syndrome

This syndrome appends when you don’t have an aggregated view of all these testing activities. In such a case, testing gaps/holes join their force to create tunnels where bugs and regressions can stay hidden until production.

Furthermore, if you map testing gaps and code modifications it represents high risks areas where new bugs or regressions will pop up.

Mike Cohn developed, in his book “Succeeding with Agile”, the Test Pyramid concept to identify how to design your test automation strategy. Mike made one more comment explaining that “Testing is an investment and the investment we make at one layer should be influenced by how well testing has been done at the other layers”. However, this requires an aggregated perspective of the test coverage of all testing activities to see what previous testing activities have well covered and what’s not yet been tested.

Some medications

One of the key features of Kalistick’s Test Coverage Intelligence solution is to help teams to figure out if they are affected by this syndrome and how to treat it. I will present few key concepts.

First, Kalistick collects test coverage of any testing activities, manual, automated, from development to deployment. Second, it captures software changes in any release. Last, Kalistick platform aggregates all these data and provides a clear view of what have been tested with a focus on changes coverage and identifies where testing gaps are.

The following dashboard, for instance, distinguishes functional tests (manual & automated) and builds tests (unit & integration test done within a continuous integration platform). The blue area represents parts of the application that are not covered by any test. The red area without any green coverage focuses on code changes that are not covered by any test.

According to how large are the gaps, Kalistick Test Coverage Intelligence platform provides a functional view to assess the related business risks and to prioritize gaps to be filled firsts.

Whenever you have prioritized gaps to be filled you need to decide the kind of tests to add (Unit, Functional, Exploratory, Manual, Automated, etc.) or to execute. Kalistick provides a Test Coverage Scores™, it assesses how much changes each test will covers in order to focus on most relevant tests. For manual test, it saves days of testing and increase quality at the same time. For automated tests it accelerates the software delivery process and the feedback to the team.

Access Kalistick platform to have a look at our dashboards (use demo/demo as login/password).

Conclusion

Swiss cheese large holes mean pronounced flavor so this is not a problem except that it doesn’t slice well and comes apart in mechanical slicers. However with software, large testing gaps means bad smells and a high probability that the application falls apart when going live.

Want to meet us?

Kalistick will be at the Eurostar Conference 2012 to present its Test Coverage Intelligence solution, Sylvain François speaks at Grenoble Agile 2012, Marc Rambert speaks at Belgium Testing Days 2013 in Brussels and  Swiss Testing Days 2013 in Zurich, and our subject propositions are still in discussion for Star East 2013 in Orlando FL  and JFTL 2013 in Paris.

The testing pyramid

For those who aren’t familiar with the testing pyramid (or who don’t remember), the principle is as follows:

Mike Cohn

The bottom tier has to be automated the most: it gives the best ROI as unit tests can be easily written by developers and gives them an immediate feedback. The more automated unit tests, the stronger will be the foundation.

The middle tier is more difficult to automate than the foundation and tests at this level will be focused on business-facing tests. These functional tests will give an answer to the question “are we building the right thing”. It will ensure functionalities match the requirements.

The top tier will the least automated tests because this level is usually composed of UI tests that are expensive to write or maintain and that often require human control (over visual aspects for instance).

What the pyramid misses

I think the pyramid is very useful and test teams can gain efficiency if they follow its principle. However, something is missing…

Tests in this multi-layer system are being executed by different people, and all these tests are of different nature. This introduces one more difficulty: there is no aggregated vision of all these different tests. For instance, it is hard to get a  complete picture of acceptance and GUI tests.

As a result, teams can leave “testing holes” within their application: software areas that are not covered by any tests and it is more likely that bugs will slip through the net of your testing in these areas!

Unknown risk coverage

However it is almost impossible to reach 100% test coverage of the software. So how to assess risks related to these “testing holes” in order to decide which tests should be added?

We suggest 2 ways of doing this assessment in order to have the right focus for additional tests.

1. Focus on regression risk

The point is to compare the current version of the software you have to test with the previous one. This comparison will allow you to identify changes where regression risks are higher.

Mixing this information with test coverage clearly identifies regression risks that have not been covered by any test.

As a picture is worth a thousand words, let me show you that new principle:

Known risk coverage

2. Add business perspectives

Even with a focus on untested changes this could still be too large when you haven’t yet got a high proportion of automated test. So in addition, we recommend adding a business perspective: drill-down to untested changes by features or  software functional areas. So you can focus your additional testing activities where bugs should be avoided in order of priority.

Functional view

Ok, but what do we do when we don’t have a high number of automated tests?

Let’s jump to the top of the pyramid, this is where you will need to have more manual tests if your coverage by automated tests is weak. In such a situation you won’t have enough time to execute all the manual tests
every time. You will have to select relevant tests and execute them depending on their priorities and on what has not been tested at lower levels.

Combining changes, coverage and business perspectives, not only will you optimize the time you spend on testing by balancing automated and manual tests, but you will also focus on relevant tests (and thus avoid losing time with useless  tests) and ensure you covered the risks in the application.

Adding test footprints

During testing, Kalistick’s agent can record all the execution paths through your application. This is not about record/replay but linking test cases and code so that when changes are made in a release, relevant test  cases to cover the corresponding regressions risks can be identified.

As you can imagine, the technology is self-learning. The more tests you execute, the better the software is going to know yours and be able to help you to build more efficient regression test strategy.

The goal of this technology is double:

1. Optimizing test efficiency by selecting relevant tests for a test campaign and avoid losing time with useless tests which aren’t testing changed modules
2. Ensuring that all risks are covered, even indirect risks dues to indirect impacts of certain changes (mainly regressions thus)

Conclusion

On your way to reach the right balance of automated and manual tests at each level of the pyramid this technology will help you improve your software testing efficiency.

An idyllic vision of test automation

Automated testing is undoubtedly an ideal solution for having the capacity of replaying test for each version of an application, and thus avoiding regressions. First steps with automation tools are generally awesome, but…

A reality not so bright

It appears that this “ideal” is not always fulfilled. Depending on projects and contexts:

• Application’s functionalities and UI are not stable enough. They change with each version and thus require manual intervention on automation scripts. Maintenance costs of test scripts become too important.
• Data are not stable on a long term basis and it is hard to control results. Analyzing every result requires as much time as executing the test.
• Visibility on real test coverage is poor. We’ve got automated tests, but knowing if they are relevant regarding to regression risks related to the evolutions is very different. Basically, you play the tests, but it doesn’t avoid regressions.

So, here is the reality as seen with our clients: there sometimes are automated test. When there are, it’s no more than 20% to 40% of tests overall!

For manual tests remaining, an efficient test strategy is essential. Available resources and time are impeding testers from executing all the tests.

How to define a regression test strategy

Identification of regression risks

It is essential to have a clear vision of risks in order to define an efficient test strategy. Otherwise, selecting relevant tests is extremely hard.

However, this clear vision is not easy to obtain.

As explained in a previous article about misunderstanding between developers and testers, developers are failing to provide test teams trustable information. Thus, no way to surely identify impacted functionalities nor risks.

Our technology is taking a picture of each version of the application. It then compares each picture to precisely identify changes and their functional impacts. Thanks to that, we you can get a clear picture of risks to cover for each version, whether major of minor.

Identifying the right test cases

Beyond risks identification, it’s necessary to select test cases that effectively cover those risks. And selecting them becomes more complex with an increasing number of tests.

Our technology is unique. For each test execution, its real footprint on the application is recorded. We accurately identify what’s tested in the application (executed code) by one test. We called it “Test Learning System”, and it’s automated.

By correlating this footprint with version analysis, our platform can identify which scenarios are relevant for covering the regression risks.

Prioritizing tests regarding business issues

Identifying tests is a good point, but it can also be necessary to prioritize them.

When time runs out and you can’t play all of the tests, better to be sure you started with the good ones. This is Risk Based Testing.

By identifying each functional module of the software, Kalistick’s platform allows to associate a business criticality level. It is then crossed with footprints and modifications in order to select and prioritize test scenarios.

An integrated wizard also allows polishing the strategy until the number of scenarios can be played in the allocated time with the available testers.

Trustworthy test coverage indicators

During the test stage, Kalistick’s platform brings a detailed vision of the risk coverage progress by identifying tested and untested elements. Thus, it becomes possible to identify the eventual falls through the net and to stitch them up before going live to production.

This vision of coverage highlights complementarity between automated and manual tests. After an automated test campaign, it allows to identify remaining parts to be tested as well as the most relevant manual tests.

In conclusion, it becomes possible with Kalistick to define more efficient regression test strategies that are balancing time, costs and risk coverage, and to adapt them to every business context, project or version.