Differences with an OpenSource integration

We are sometimes asked if our solution, the Cockpit, is different from existing OpenSource tools. Our best response generally involves providing a demonstration of our tool, but the objective here is to make our arguments more specific.

Everyone knows now that installing OpenSource tools eventually reveals hidden costs. There have been many marketing discussions on this topic, supporting the position of proprietary tools, with a sometimes significant expression of FUD.

Our platform is itself based on an infrastructure that includes OpenSource products, and we are well-positioned to know that OpenSource is an often relevant alternative. We say this simply to emphasize the complexity of implementing a quality tracking tool using currently available OpenSource products. The purpose is to show that the effort resides not only in installing the tool, but also in the additional tasks that are needed in order to make a true quality improvement.

Selection of tools

Selecting tools is obviously critical to how the remaining steps work. The supply of quality-related tools presents some particularities:

• There is an abundant amount of tools. There are several dozen tools for various languages. For example, here is a non-exhaustive list: http://www.laatuk.com/tools/review_tools.html.
• Tools are difficult to compare. Tools that measure metrics, for example, offer their own set of metrics. And if some metrics are split between different tools, they are not always calculated the same.
• Because quality analysis is a highly popular subject of research, there are many tools being developed by universities. These tools sometimes stop being developed once the corresponding thesis is complete or cannot be adapted to practical use on professional projects (some examples: ckjm, JMT, iPlasma, etc.).

Installation and integration

Installing these tools presents a level of complexity that varies greatly from case to case. Installation is usually quite simple. Only mashup tools, which combine different measurement types, sometimes require real work to configure or install third-party tools (see XRadar, QALab, etc.).

We should note two things:

• Using these tools is only effective when they are standardized, such as within a continuous integration process. The existing procedure must therefore be modified to launch these tools and to publish their results so that the entire team can see them.
• For a full view of the quality of development, often several tools are aggregated, each providing different measurements. It is therefore difficult to get a consistent summary from these heterogeneous results. Some mashup tools offer a first level response to this problem.

Finally, we have to consider how these tools are updated or how they would be used to resolve bugs or add new features or rules.

Project settings

Besides the installation and configuration that is inherent to the tool, we must also configure the scope of code to be analyzed. Depending on the tool, it may work with source files and/or compiled files. We should also be able to supply the third-party libraries that are used for compiling and/or executing the code (which varies according to the tool). Finally, we should be able to exclude some of the code from the analysis: generated code, patched third-party code, test code, etc.

This work should be done at the start of the process, and it continues throughout the project and requires expertise to be transmitted whenever the team changes. The ease of updating the settings should be an essential factor when selecting tools. No matter what, a development must always be well structured in order to be automatically analyzed.

Rule Settings

The most critical step involves selecting the quality rules. Some tools provide several hundred rules. It is rarely worthwhile to keep all of these rules because developers will soon be annoyed by an avalanche of anomalies, so we should work with a selection. But this job is complex:

• It required true expertise concerning the technology used.
• A result depends greatly on its implementation. Whether a bad practice, a metric, or a copy/paste, the ways they are implemented are high variable, based on the tools. We must therefore experiment with the tool before determining whether any particular rules are truly relevant for a project.
• As for metrics, setting thresholds is always a delicate process. For example, what should we choose as the upper threshold for the cyclomatic complexity of a Java method? And a C# method?
• This selection work will likely stir up debates within the team over discussed practices, thresholds thought to be too demanding, etc. It should therefore be handled by someone who knows the subject perfectly.
• Existing code is particularly difficult to deal with. How do we ensure the quality of development problems without making code unstable that has already passed functional testing? This issue is discussed in another entry.

In all cases, the chosen rules must be ranked according to their priority or severity in order to make processing easier.

Training and building awareness

In order to correct problems and be able to avoid repeating them, developing should be trained on the chosen quality rules. There are two possible strategies:

• Review the rules upstream in the project via an oral presentation or an exhaustive recommendation guide.
• Let the developers learn about the quality rules by consulting the analysis results. The developers should then have easy access to the documentation for each rule, ideally with examples and links. The true objective is not just to correct problems, but to expand expertise so as to not repeat problems, thus having a truly continuous improvement process.

In practice, we find that the first solution is not very effective because the developers only retain some of the rules they are shown. Our customers' experience shows that the second solution seems to be more effective, combined with a general presentation upstream to show some structural rules that will be applied with the early developments.

Correction plan

Detection of violations of quality rules is the first step, but the objective is to correct these anomalies. This aspect is neglected by most tools, even though it is fundamental for quality monitoring to be effective and for the team to be able to measure the application of the quality rules.

A correction plan might involve an allotment of points to be corrected: by priority, by developer, by type, etc. Why not handle these corrections like tasks to be completed, like in a bug tracking tool? This allows for better integration of the correction process in the schedule and tracking corrections that have been made.

Conclusion

Implementing a quality monitoring process using OpenSource tools is a real alternative, but be sure to thoroughly compare the items needed in order to get effective results. Besides showing that the project has this or that process, the objective is to achieve a tangible return on investment.

A Cockpit for controlling quality

This post starts our blog, which is designed for sharing our experience with quality in C# and Java development projects. This experience comes from members of our R&D team and from what we have accumulated each day on our Cockpit platform from the millions of lines of code that have been analyzed in our customers' projects.

We created Cockpit after having found that no tool controlled Java or C# development quality as simply and pragmatically as we wanted. The is a huge potential for improvement in the field of development technical quality. Generally speaking, there are two major categories of tools on the market:

• Analysis agents, often OpenSource for Java environments, that typically specialize in a certain type of detection (metrics, style, bad practices, duplicated code, etc.) for a particular language.
• Products, almost always commercial products, that combine the different measurement types and handle several languages in very different ways (object, procedural, query languages, etc.).

The first are directed mainly at technical profiles: developers, architects, technical project managers, etc. Their implementation may or may not be complex, but the effort tends to include the following steps: setting the rules and thresholds, integration into an automated process, training and raising awareness among development, definition and monitoring of an improvement plan for correcting problems, configuration maintenance, and updates to the tool. I will go into detail on all of these in a future post.

As for the commercial products, the two obstacles we have identified and have confirmed with our customers are the cost and complexity of using them. Products of this type usually have a very high licensing cost (well over €10,000), plus the possible installation, configuration, updating, advisement, and support fees. Few projects have the necessary budgets for this. The cost can often be explained by the rich features offers to the user or by how flexibly it can be set up. But these characteristics also make using the product more complex.

In both cases, many implementation attempts have failed for various reasons:

• Developers could not have enough control over the indicators.
• They felt as if they were flooded with alerts.
• The information provided is too technical. It does not provide an understandable view of quality for the project managers and managers who are not involved in the project management.
• Everyone has difficulty measuring the return on investment for this quality process.

In these conditions, monitoring quality seems more like a constraint than an way to help improve the project.

This may seem extreme, but many are surely satisfied with the tools they have adopted and consider them to be effective. Yet it seems common, which is why we came up with a different approach to quality, one that can truly be perceived as positive by everyone involved with the project, not just by a single theoretical gauge:

1. Monitoring quality must be simple. This is why we have chosen a SaaS model, where the user installs nothing on their own infrastructure and can start their first analysis within a few hours. This monitoring should be included in the company's development process (agile, V cycle, continuous integration, etc.).
2. Monitoring quality must be pragmatic. Quality requirements must be adapted to the needs and context of the project. Not all projects have the same quality requirements, and overquality is costly. It is better to work toward a realistic objective that the team can achieve, perhaps by refining requirements along the way.
3. The various people involved with the project must share the same view of the status of quality. For this, the information should be presented differently depending on the person viewing it, such as a project director or a developer, but it should express the same results. Using an external third-party service also makes it easier to adopt this common view.
4. Quality rules should be up to date and adapted to the current status of the application. Development technologies change (languages, frameworks, etc.), and rules should follow suit. Having a statistical database that is updated daily from the analyzed project code seems to us to be a key point in guaranteeing an updated and relevant rule repository.

That introduces our approach. We have been working on our technology for three years. We have been able to validate our approach, validate its use by our customers, and now we can share our results with you. The next few posts will be more concrete, dealing more directly with this feedback.