Quality Assessment

TIOBE offers a software quality assessment service based on the official ISO/IEC 9126 standard on software product quality. The result of the assessment is a report and a presentation. The report lists all findings and explains what the impact of these are on the software quality of your system. Moreover, it benchmarks the results with the industrial standard and concludes with a list of pragmatic recommendations. The assessment ends with an on-site presentation to explain the findings to all stakeholders.

An example of the high level results of an assessment is:

Assessment Area Value
Reliability +
Maintainability 0
Total 0/-

There are so many software quality assessments available nowadays. What makes us special? The most striking difference is that our starting point is the source code, and not the process or the available documentation. The reason for this is that source code is the only tangible asset in a released software product. Once code is shipped, it doesn’t matter any more how nice your architecture documents were or how well-defined your software development processes.

A positive side-effect of taking source code as starting point is that most work on code quality can be automated. This means that the results can be obtained fast and in a relatively inexpensive way. Moreover, the generated data is reproducible and can be benchmarked.

The following 3 ISO/IEC 9126 quality factors are measured in detail:

Reliability. Reliability is about the stability of a system. How does it react in undefined situations? How often does it crash? This quality attribute is measured by identifying occurrences of “null dereferences”, “array out of bounds”, and “division by zero”. These kind of crashes are detected automatically by applying state-of-the-art code checkers. Apart from this, also the code coverage of the performed tests is inspected. Code coverage is measured with appropriate test coverage tooling. The following picture shows the test coverage for a software system.

Test Coverage

Changeability. How much effort does it cost to extend an existing software system? The cost will be higher in case a system is hard to change. Measuring changeability comes down to determining the modularity and re-usability of a software system at all levels. Examples of metrics that are applied here are “cyclomatic complexity”, “fan-in/out” and generic architectural rules, like. layering. In the picture below the high level relations between software modules is displayed, which is input for the changeability analysis.

Module Relations

Maintainability. A well maintainable software system is comprehensible, thus preventing the introduction of new programming errors while making changes. Moreover, it is easier to let others (internal or external to your company) take over. The most effective method to calculate maintainability is to apply all kinds of coding standards to the software. TIOBE has developed a special metric to indicate how compliant a software system is to (a set of) coding standards. this metric is called the confidence factor. The confidence factor for a software system is shown below.

It is important to note that other ISO-related quality factors or company specific requests can be measured as well during the assessment. Examples are the identification of performance bottlenecks or judging the longevity of external libraries.

Confidence Factor

TIOBE’s quality assessment consists of 4 basic packages. All these packages measure “reliability”, “”changeability”, and “maintainability”, but each in a specific area. The more packages are applied the more accurate the final results are.

All packages are optional (except for the first one). TIOBE also offers some extra non-standardized packages. These are “deployability”, “infrastructure”, “build processes”, “software processes”, and “business risks”.

Assessments are conducted by top experts in the field. For this reason, TIOBE subcontracts external consultants for some of the topics. The following companies help TIOBE to get the best results:

Topic Company
Web services

Besides this human expertise also state-of-the-art tools are used to analyse software code automatically. These include Lattix (Lattix company), Jtest (Parasoft), SourceMonitor (Campwood), and SOAtest (Parasoft). Please contact us in case of any questions: sales@tiobe.com.