How to select Code Analysis Tools

Code checking tools are software products that check compile-time whether a software program adheres to a certain set of rules of a coding standard. Without such code checkers, the enforcement of a coding standard in an organization is likely to fail. There are two main causes for this: (1) the number of rules in a coding standard is usually so large that nobody can remind them all (except the author) and (2) some context-sensitive rules that demand inspection of several files are very hard to check by human beings.

Although code checkers can reduce formal code review time considerably, it certainly does not mean that code reviews are not needed anymore. Some coding rules cannot be checked at all (e.g. "Comments should be in English") or only partly at compile-time (e.g. "Use delete, whenever you use new"). On average about 65% of a coding standard can be covered with code checking tools.

There are several characteristics that should be assessed during the selection process of a code checker. There are 6 issues that are most important. Obvious requirements such as whether it is available for the used platform are not taken into account.

Availability to define own rules. Experience has shown that only about 20% of a propietary coding standard is available as built-in rule in a code checker. As a consequence, there should be a possibility to define own rules. This can be via an API (complex in use but powerful) or a graphical user interface (easy to use but limited).

Integration in programming IDE. Most end users of code checkers are software engineers. Therefore a code checker should fit smoothly in the used programming environment. Preferrably via a toolbar with an easy way of running the tool (one button push).

Presence of command-line version. In order to be able to integrate the code checker in the software development process and/or in the programming IDE (if no plug-in is available) it should be command-line oriented. Command-line versions also allow for nightly batch runs of the code checker to collect quality data.

High performance. Code checkers are slower than compilers by nature because they perform a more in-depth semantical analysis of the source code. However, performance is a key issue for end user acceptance of the tooling. Response times of more than 1 minute per file are perceived as not workable.

Low cost of ownership. Cost of ownership means purchase price and maintenance costs. Since code checkers are not used continuously by end users, floating licenses appear to be more cost effective in most cases. About 1 floating end user license is needed per 10 developers.

Available support. Decent product support is an essential feature of a code checker. Good support does not only mean prompt and adequate response to user questions and problems, but also the frequency of new product release. "We will solve this in the next release" is not interesting if the average product release cycle is 1 year.

This How-to is an excerpt from a article called "Industrial methodology to introduce coding standards" and i feel the considerations and points are logical and good to follow for the code analyzers evaluation.