Tag Archive for code quality

10 Code Coverage Tools for C & C++

Code coverage is a measure used in software testing that describes the degree to which the source code of a program has been tested. It a form of white box testing as it is a form of testing that inspects the code directly.

Software_testing

Image Source : http://www.zeeteenet.com/images/sw_testing.jpg

Recommended Reading:

To measure how well the software is tested by a test suite, one or more coverage criteria are used. There are a number of coverage criteria and here are the important ones:

Function coverage – that says if each function in the program been executed or not
Statement coverage – that says if each line of the source code been executed or not
Decision coverage (also known as Branch coverage) – that says if each control structure such as an if statement has been evaluated both to true and false
Condition coverage – says if each boolean sub-expression has been evaluated for both true and false
Path coverage – that says if every possible route through a given part of the code been executed or not
Entry/exit coverage – that says if every possible call and return of the function been executed or not

In this article, we cover the popular Code Coverage tools to measure Code Coverage.

1. Testwell CTC++

CTC++ is a powerful instrumentation-based test coverage and dynamic analysis tool for C and C++ code. As a coverage tool, CTC++ shows the coverage all the way to the Modified Condition/Decision Coverage (MC/DC) level. As a dynamic analysis tool, CTC++  shows the execution counters in the code, i.e. more than plain boolean coverage information. You can also use CTC++ to measure function execution costs (normally time) and to enable function entry/exit tracing at test time

2. CoverageMeter

CoverageMeter is a complete code coverage tool chain for C/C++ programs available under Linux or Microsoft Windows. It analyzes the performance of a software validation and permits to measure the performance and optimizes the testing process of a C or C++ applications by:

  • Finding untested code sections.
  • Reducing the amount of tests by finding redundant tests – With CoverageMeter it is possible to find which portion of the source code is covered only by one execution of a test, and to detect if a new test does not cover more source code line than existing tests.
  • Finding dead code trough displaying the code parts which are never executed.
  • Specially useful for manual testing – CoverageMeter is able to calculate the optimal execution order of tests which maximize the overall coverage after each run.
  • Also, CoverageMeter is able to perform its analysis on a difference of two applications.
  • This permits to find which tests are impacted by a source code modification and permits to measure the test quality of a patch or a hot fix.

It is composed of 4 tools:

With CoverageScanner you can analyzes, instrument and generate the C/C++ application. CoverageBrowser displays and manages the results of the coverage analysis. Execution Agent is a small free tool which is dedicated to manual tests and specially for black box interactive tests. It also has an optional Microsoft Visual Studio 2005 & 2008 Add-In which permits to generate code coverage configurations from every C/C++ projects created by Microsoft® Visual Studio.

3. BullseyeCoverage

BullseyeCoverage is a code coverage analyzer for C++ and C that tells you how much of your source code was tested. You can use this information to quickly focus your testing effort and pinpoint areas that need to be reviewed. Here are some of the features of BullseyeCoverage:

  • Function coverage gives you a quick overview and condition/decision coverage gives you high precision
  • Works with everything you can write in C++ and C, including system-level and kernel mode
  • Short learning curve
  • Integrated with Microsoft Visual Studio
  • Include or exclude any portion of your project code
  • Run-time source code included, for custom environments

4. GCT

GCT is a C Code Coverage Tool. GCT was the third coverage tool by Brian Marick. It instruments C code in a source-to-source translation, then passes the instrumented code to a compiler. It is suitable for measuring the coverage of embedded systems as it was first used on a Unix kernel. In addition to branch and multiple-condition coverage, it also has boundary-condition and loop coverage. For more details on this tool check the tutorial at testingcraft.com

5. CppUnit

CppUnit is a unit testing framework module for C++, described as a C++ port of JUnit. The library is released under the GNU Lesser General Public License

6. Dynamic Code Coverage

Dynamic Code Coverage identifies untested portions of your software code, so your team can focus attention on those undiscovered areas, and improve your software quality immediately. Dynamic Code Coverage builds a detailed coverage analysis, covering every function, line, decision, and branch. Coverage files from subsequent multiple runs get added in. You can look at the information you gather in a variety of ways. And you can annotate each source file with function, line, decision, and branch information.

7. TCAT C/C++

TCAT C/C++ for Windows measures the Effectiveness of Tests, Test Suites. It identifies untested code and improves test efficiency.

It provides a combined branch and call-pair coverage for C-C++. It provides annotatable calltree displays and digraph displays with access to source.

8. COVTOOL

COVTOOL is an open source test coverage analyzer for C++ programs. It lets you dynamically instrument your source code as you compile. An instrumented program keeps track of the lines of code that were executed during its run and produces a log of the same upon program termination. Multiple program runs will produce multiple logs.
You can then use the many log files generated during a suite of tests to analyze test coverage percentages. Most importantly, you can use the coverage information to annotate your source. Thus you can find which lines in which source files did not get executed during your entire suite of tests. Code that has been tested may still be buggy, but you should have almost no confidence in code that is never tested.

9. gocv

gcov is a tool you can use in conjunction with GCC to test code coverage in your programs. You can use it with along with GCC to analyze your programs to help create more efficient, faster running code and to discover untested parts of your program. You can use gcov tool to help to understand where your optimization efforts will best affect your code. You can also use gcov along with the other profiling tool, gprof, to assess which parts of your code use the greatest amount of computing time.

10. xCover

xCover is a Code Coverage library for C and C++, written in C. xCover itself is a platform-independent library. It has been tested and used on both UNIX and Windows – in principle, it should work on any operating system for which a compatible C/C++ compiler exists.

12 Source Code Profilers for C & C++

Code Profilers are very distinct from traditional debuggers. They are able to catch the trivial and non fatal coding errors which are often hard for humans to catch. These trivial bugs later claim weeks of debugging and extra cost to fix. Code profilers will save you hours of debugging time. These tools can automatically detect many memory management and threading, etc bugs. These tools will give you confidence that your programs are free of many common bugs, some of which would take hours to find manually, or never be found at all. You can find and eliminate bugs before they become a problem.

As usual, there are a lot of code profilers available for C/C++. Here is a list of very prominent such players.

1. Insure++ (For Windows, Linux, UNIX like)

image

Insure++ is a runtime memory analysis and error detection tool for C and C++ that automatically identifies a variety of difficult-to-track programming and memory-access errors, along with potential defects and inefficiencies in memory usage. Errors such as memory corruption, memory leaks, access outside of array bounds, invalid pointers, and the like often go undetected during normal testing, only to result in application crashes in the field. Insure++ will help you find and eliminate such defects in your applications to ensure the integrity of their memory usage.

During testing, Insure++ checks all types of memory references, including those to static (global), stack, and shared memory — both in user’s code and in third party libraries.

Errors that Insure++ detects include: Corrupted heap and stack memory, Use of uninitialized variables and objects, Array and string bounds errors on heap and stack, etc.

2. Rational Purify (For Windows and Linux/UNIX)

image

IBM Rational Purify is a runtime analysis solution designed to help developers write more reliable code such as C++ and C. Reliability is ensured via two crucial functions: memory corruption detection and memory leak detection. Rational Purify packages support for these two runtime analysis capabilities in a single product with a common insta ll and licensing system.

3. Valgrind (For UNIX like)

Valgrind is an award-winning instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.

The Valgrind distribution currently includes six production-quality tools: a memory error detector, two thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache profiler, and a heap profiler. It also includes one experimental tool, which detects out of bounds reads and writes of stack, global and heap arrays. It runs on the following platforms: X86/Linux, AMD64/Linux, PPC32/Linux, PPC64/Linux.

Valgrind works with programs written in any language. Because Valgrind works directly with program binaries, it works with programs written in any programming language, be they compiled, just-in-time compiled, or interpreted. The Valgrind tools are largely aimed at programs written in C and C++, because programs written in these languages tend to have the most bugs! But it can, for example, be used to debug and profile systems written in a mixture of languages. Valgrind has been used on programs written partly or entirely in C, C++, Java, Perl, Python, assembly code, Fortran, Ada, and many others.

Valgrind is Free and Open Source Software. Read here for more on Valgrind.

4. Zoom (For Linux)

image

image

Zoom is a graphical, low-overhead, sampling profiler for Linux from RotateRight which diagnoses performance problems related to CPU and memory-intensive code. Zoom supports Call Tree Visualization, Code Visualization etc. It is really easy to correlate assembly and actual source, bottlenecks in the code etc. Zoom supports DWARF (debugging data format).

5. Code Analyst (For Windows and Linux)

image

AMD CodeAnalyst Performance Analyzer is a suite of powerful tools that analyze software performance on AMD microprocessors and help developers optimize application performance. It uses a profiling technique to identify performance areas and issues within the system. Using this suite of tools, developers can get visibility into overall system performance, and can navigate into different modules and functions to locate hotspots within the target application.

CodeAnalyst allows you to profile applications using various performance events available on different AMD processors. You can specify performance events to target various functional units of the processor, or profiling granularity to adjust the accuracy of the result.

CodeAnalyst is designed to profile the performance of binary modules, including user mode application modules and kernel mode driver modules. Timer-Based Profiling and Event-Based Profiling collect data from multiple processors in a multi-processor system. It can support any language as it works with the binary.

6. Sleepy (For Windows)

image

‘Sleepy’ is a non-invasive profiler for the Windows platform that allows statistical profiling of C++ applications through sampling the instruction pointer register. Sleepy presents results in a GUI built with wxWindows.

The Sleepy profiler uses a technique where the profiler runs in a different thread from the target program. Every 1ms or so, the profiler thread suspends the target thread, and pulls out the current instruction pointer register value from the thread context. These mem addresses are resolved into procedure names and line numbers using debug information. This allows line-level resolution, without making any changes to the target program. The only requirement is that the target program is compiled with (MS) debug information

Sleepy uses the MS Debug Help (dbghelp) library to read debug information, and is therefore limited to the debug info formats that dbghelp can understand. As far as I know, debug information produced by non-MS compilers such as GCC will not be understood, and therefore such code cannot be profiled.

An improved version Very Sleepy is also available.

7 Intel VTune (For Windows and Linux)

image

VTune Performance Analyzer is Intel Corporation‘s tool for call graph or analyzing a set of tuning events. It works with C/C++/Fortan/.NET/Java and other applications on Linux or Windows, but only when running on selected Intel hardware. he GUI assists in various kinds of code profiling including time based profile, Event based profile etc.

VTune Performance Analyzer evaluates applications on all sizes of systems based on Intel® processors, from embedded systems through supercomputers, to help you improve application performance. VTune Performance Analyzer makes application performance tuning easier and is indispensable for making your software run its fastest on the latest single and multi-core systems.

8. AQTime (For Windows)

image

AQtime is a complete, top-to-bottom performance profiler and memory usage debugger for all major Windows development tools (Delphi, Visual Basic, Visual C++, and C++Builder), .NET compilers (including both Microsoft and non-Microsoft) and for VBScript and JScript routines. It includes several profilers and productivity tools that provide you with an unrivaled level of information on the state of your 32- and 64-bit software projects – from inception to delivery!

9. BullseyeCoverage (For Windows, Linux, Solaris and many more)

image

BullseyeCoverage is a code coverage analyzer for C++ and C that tells you how much of your source code was tested. You can use this information to quickly focus your testing effort and pinpoint areas that need to be reviewed. Code coverage analysis is useful during unit testing, integration testing, and final release. BullseyeCoverage enables you to create more reliable code and save time.

Function coverage gives you a quick overview and condition/decision coverage gives you high precision Works with everything you can write in C++ and C, including system-level and kernel mode Short learning curve, Integrated with Microsoft Visual Studio, Include or exclude any portion of your project code, Run-time source code included, for custom environments, Simple licensing, Merge results from distributed testing etc.

10. C++ Profiler Tool (For Windows)

image

The C++ Profiler tool enables the collection and display of execution profile data on C++ software source code bases of arbitrary size.

11. Shiny Profiler (For Windows, UNIX Like, Source code is portable)

Shiny is a low overhead, well documented and lightning fast C++ profiler. Easy to use on existing projects, no extensive surgery, just have fun with it. Nicely formatted output provides call tree (call graph), flat profile and is simple to custom display

12. Deleaker (For Windows)

image

Deleaker is a run-time error detection and debugging tool for Visual C++ developers. Deleaker is a really useful add-in for Visual Studio that helps you to analyze programming errors, many of which are unique to Visual C++. With Deleaker you can detect and localize resource leaks such as memory, GDI and USER objects, handles.

Resource leaks. Even a robust Windows-based application is not safe from them. Of all bugs they are the most difficult to find especially when they appear in graphics device interface (GDI) objects and menus. And as we all know, the earlier you catch bugs, the less expensive they are. Plenty of tools exist to help track down memory leaks. But there are very few good tools to help track GDI resource leaks that can ruin performance on any Windows operating system. Deleaker is one of them and the fact that it almost doesn’t slow down your application makes it your natural choice.

Open Source Projects with Exceptionally Low Defect Density

Scan site along with Coverity Prevent, did an objective measure of static analysis defect density in popular Open Source projects. The analysis is based on the fact that previous reports from Microsoft has proven that static analysis can be an accurate predictor of defect density ( Source: Static Analysis Tools as Early Indicators of Pre-Release Defect Density, Microsoft Research ).

Static Analysis Defect Density is defined as the number of Coverity Prevent discovered defects per 1,000 lines of code in a given project or set of projects.

As part of this research, Coverity Prevent™, the industry leading static analysis tool, was made available to qualified open source software projects via the Scan website. Through the Scan site, open source developers can retrieve the
defects identified by Prevent analyses through a portal accessible only by qualified project developers. scan.coverity.com

By comparing the number of defects identified in the first analysis of each open source project to the number of defects found in the most recent analysis, Coverity measures the overall progress of participating open source projects at the Scan site.

Change in Defect Density Across All Open Source Projects

Based on the Scan 2006 Benchmark, the initial static analysis defect density averaged across participating projects is 0.30, or roughly one defect per 3,333
lines of code.

The current average number of individual defects per project, based on the Scan 2006 Benchmark (as of March 2008) is 283.49. Based on the consolidated results of the most recent analysis for each project, the current static analysis defect density averaged across all the participating projects is 0.25, or roughly one defect per 4,000 lines of code.

These findings represent an overall reduction of static analysis defect density across 250 open source projects of a total of 23,068 individual defects, lowering the average static analysis defect density in these open source projects by 16%.

Average Number of Defects

Initial Static Analysis Defect Density-Current Static Analysis Defect Density-

Certainly, there is a change in the defect density across various open source project. The findings represent an overall reduction of static analysis defect density across 250 open source projects of a total of 23,068 individual defects, lowering the average static analysis defect density in these open source projects by 16%.

Change_in_defet_density1

Frequency of Individual Code Defect Types

To provide insight into general trends regarding the frequency of specific defect types, consolidated totals across all open source projects are presented in the list table

  1. NULL Pointer Dereference: Number of Defects: 6,448 Percentage: 27.95%
  2. Resource Leak: Number of Defects:5,852 Percentage: 25.73%
  3. Unintentional Ignored Expressions: Number of Defects: Percentage: 2,252 9.76%
  4. Use Before Test (NULL): Number of Defects: 1,867 Percentage: 8.09%
  5. Buffer Overrun (statically allocated): Number of Defects: 1,417 Percentage: 6.14%
  6. Use After Free: Number of Defects: 1,491 Percentage: 6.46%
  7. Unsafe use of Returned NULL: Number of Defects: 1,349 Percentage:5.85%
  8. Uninitialized Values Read: Number of Defects: 1,268 Percentage:5.50%
  9. Unsafe use of Returned Negative: Number of Defects: 859 Percentage:3.72%
  10. Type and Allocation Size Mismatch: Number of Defects: 144 Percentage: 0.62%
  11. Buffer Overrun (dynamically allocated): Number of Defects: 72 Percentage: 0.31%
  12. Use Before Test (negative): Number of Defects: 49 Percentage: 0.21%

Projects with Exceptionally Low Defect Density

The site divides open source projects into rungs based on the progress each project makes in resolving defects. Projects at higher rungs receive access to additional analysis capabilities and configuration options. Projects are promoted as they resolve the majority of defects identified at their current rung.

Rung 0

The first rung is rung 0. At rung 0, a project has been built and analyzed by Coverity’s Scan infrastructure, but no representatives of the open source project have come forward for access to the analysis results. Projects progress to the next rung by selecting a set of official contacts to represent the project to Coverity

Currently there are 173 projects at Rung 0 – http://scan.coverity.com/rung0.html

Rung 1

The next rung is rung 1. At rung 1 and above, Coverity supplies a mailing list for developers to discuss analysis results, and to facilitate communication from Coverity about questions from the project or additional functionality being made available. Projects progress to the Rug 2 by reaching a reasonably low defect count in the basic issue types, appropriate for the size of the project code base.

Currently, there are 86 projets in Rung 1 – http://scan.coverity.com/rung1.html

Rung 2

The following projects are the projects with exceptionally low defect density and have advanced to Rung 2 of the Scan ladder.
Amanda
• NTP
OpenPAM
OpenVPN
• Overdose
• Perl
• PHP
• Postfix
• Python
Samba
• TCL

For details on defect density check http://scan.coverity.com/rung2.html

All of these projects eliminated multiple classes of potential security vulnerabilities and quality defects from their code on the Coverity Scan site. Because of their efforts to proactively ensure software integrity and security, organizations and consumers can now select these open source applications with even greater confidence.

Amanda’s developers fix over 40% of the Scan’s detected defects with a single reading of the Scan analysis for that issue. In the chart below, the red defects were RESOURCE LEAKs.

Amanda Chart

Over 75% of the defects Scan identified in Samba were fixed within two reviews of the Scan analysis. In the chart below, the blue defects were NULL DEREFERENCEs

samba_Chart1

Conclusion

Findings are based on analysis of over 55 million lines of code on a recurring basis from more than 250 open source projects, representing 14,238 individual project analysis runs for a total of nearly 10 billion lines of code analyzed:

The overall quality and security of open source software is improving – Researchers at the Scan site observed a 16% reduction in static analysis defect density over the past two years
Prevalence of individual defect types – There is a clear distinction between common and uncommon defect types across open source projects
Code base size and static analysis defect count – Research found a strong, linear relationship between these two variables
Function length and static analysis defect density – Research indicates static analysis defect density and function length are statistically uncorrelated
Cyclomatic complexity and Halstead effort – Research indicates these two measures of code complexity are significantly correlated to codebase size
False positive results – To date, the rate of false positives identified in the Scan databases averages below 14%

Source: scan.coverity.com

Technorati Tags: , ,

Related Posts with Thumbnails