Software correctness tools should be a core component of any quality assurance (QA) program, and the Intel® Inspector XE provides functionality your QA department can use to simplify getting the answers it needs.
Tools and Options to Consider for QA
Dynamic analysis is the term that encompasses both memory checking and thread checking in the Intel Inspector XE. Use memory checking to look for memory leaks, memory corruption, allocation/deallocation mismatches, and inconsistent memory API usage, among other checks. Use thread checking to find data races (both heap and stack), deadlocks, and thread and sync API errors on threaded code.
Dynamic analysis only tests the portions of an executable that are actually invoked. Intel Inspector XE thread and memory checking tools can spot errors that could happen in an executed section of code whether or not they happen in this particular run. These tools cannot report any errors in code that was not executed.
To take advantage of static analysis, you need a full studio suite (for example Intel® Parallel Studio XE or Intel® Cluster Studio XE), rather than just the Intel Inspector XE on its own. Static analysis is implemented as a compiler-based analysis feature in the Intel® Composer XE. It detects various dangerous memory conditions - such as possible buffer overflows - that may be determined at compile time. This data is then displayed and manipulated using the Intel Inspector XE. Because static analysis covers all sources that are compiled together, there is no need to have specific data sets for analysis; however this can also make it difficult to automate focusing on particular sections of source.
For more information about static analysis, see http://software.intel.com/en-us/static-analysis
Tactics for Managing Analysis Overhead
Data set size has a direct impact on target execution time and analysis speed.
For example, it takes longer to process a 1000x1000 pixel image than a 100x100 pixel image. One possible reason could be that you have loops with an iteration space of 1...1000 for the larger image, but only 1...100 for the smaller image. The identical code paths may be executed in both cases. The difference is the number of times these code paths are repeated.
You can control analysis cost, without sacrificing completeness, by removing this kind of redundancy from your target. Instead of choosing large, repetitive data sets, choose small, representative data sets.
Data sets with runs in the time range of seconds are ideal. You can always create additional data sets to ensure all your code is inspected, or use exiting unit tests to execute sections of code in an understood fashion. When selecting your data sets, focus on code coverage in breadth, not in depth. You want to execute different paths through your code, but the specific values of variables on those paths are not important.
There are other changes you can make to reduce analysis cost, including lowering the number of threads used. For more information about things you can do to control analysis cost, see http://software.intel.com/en-us/articles/intel-inspector-xe-2013-controlling-analysis-cost
In general, you understand your code base better than anyone else, and you know where in your code you think it is most critical to focus. However, if you have a license for the Intel Parallel Studio XE, you may want to take advantage of the code complexity information available from static analysis to give you an idea of how many different paths there are through your code.
For more information on code complexity (cyclomatic complexity), what it is, and what it can mean for unit testing and quality assurance, see http://software.intel.com/en-us/articles/creating-and-viewing-cyclomatic-complexity-data-with-intel-parallel-studio-xe-2013 .
Reporting Information about New Issues Only
Whether you are using static analysis or one or more forms of dynamic analysis, you need a way to focus on new issues in your code. There are two options the Intel Inspector XE provides for getting that information. To use these methods, you need a known baseline result.
For optimal usage, of course, the baseline result should be taken from a run where you have successfully eliminated all real problems. Anything left is a positive you have determined you can just ignore. For example, you may have more than one thread accessing a variable without synchronization in between, so the thread checking tool finds a race, but you may know there are other timing factors that make it impossible for a race to occur. Alternatively, there could be an issue in a third-party library that is of no interest to you. This is particularly true for static analysis, where every source file involved in the compilation is checked, regardless of code source.
The baseline result could be generated on a per data set basis or one baseline for the entire product, depending on the method used to find only new issues. The baseline result should be generated using the same analysis level you plan to use for your testing (such as m1 for m1 testing, t3 for t3 testing).
There are three different ways to use the Intel Inspector XE to focus in on new issues:
· Suppress all
· Filter by state
· Results comparison
Suppress-all functionality is only available through the command line interface (CLI). Take the baseline result for this data set and run the following command:
inspxe-cl -create-suppression-file mySup
This creates a suppression file called mySup that contains a suppression rule to suppress every error in the most recent result. This file can be used from the CLI or the GUI .
The best way to do this is to create a different suppression file for each data set you might use to test your program. However, you can also put all of those suppression files into one directory and tell the Intel Inspector XE to use that directory to simplify your scripts. Loading too many suppressions may slow finalization of your data, so be sure to clean out any old suppression files and generate new ones if previously known issues are removed.
Filter by State
The filter-by-state method is available in the GUI or the CLI. Intel Inspector XE, by default, compares every new run of a project to the previous run and sets the state on any issue not in the previous run to New.
If you are using this for regression testing, you will want separate directory/project for each data set you plan to test and each needs its own baseline result. Intel Inspector XE will automatically use the previous result when bringing forward states. You will need to set the configuration to specify the correct baseline result before collection.
If you are using the GUI integrated into Visual Studio*, from the Visual Studio* menu, choose Tools > Options.... In the Options dialog box, expand the Intel Inspector XE 2013 folder and choose the State Management page.
If you are using the Standalone Intel Inspector XE GUI, choose File > Options.... In the Options dialog box, choose the State Management page.
Select get problem states from a specific result radio button and enter the path to the baseline results file. Once this is done you can rerun in this project and it will always use this same results file.
If you are using the CLI for your regression analysis, you will need to use the –baseline-result option to point to the correct baseline:
inspxe-cl -collect mi3 -baseline-result r000mi3 -result-dir r001mi3 -- ./a.out
Collects a mi3 run on a.out, storing it in r001mi3, and using r000mi3 as the baseline result for bringing states forward.
After running, you can use the GUI Filters pane to display just the New issues, or you can take advantage of CLI reporting to report out only the issues in the New state:
inspxe-cl -report status
Generates a status report, including state of all issues, useful for determining if there are any New issues.
inspxe-cl -report problems -f state=new
Generates a report of problems in the New state.
Results comparison is available using the Compare button in the GUI. This allows you to enter two results and get a report containing all the issues in both results and whether each problem was found in the first result only, the second result only, or both results.
You can then filter to limit to the results in the new run only, and export a report based on that.
You can achieve the same outcome in the CLI by asking for a report for two results at the same time:
inspxe-cl -R summary -r myRes000ti -r myRes001t1
Provides a summary report of the results comparison of the previously collected results found at myRes000ti and MyRes001t1
This requires you to have a baseline result for each data set you are using in your testing, although you do not need a project for each data set.
Choosing the Best Option
The suppression file for suppress-all is generated once for each data set and then that file can be used and copied wherever you want, or all of your suppressions can be co-located. However suppression of a large number of known issues can cause performance degradation.
Using compare is faster; however, it requires a separate baseline result for each data set you are using with your application.
Filtering by state is by far the fastest, as bringing forward states is already part of the default finalization; you are just choosing a baseline to bring the states from. Filtering by state, in the CLI or in the GUI, requires a separate baseline result for each data set you are using with your application.
In most cases, filter by state is probably the best option. If having a separate baseline per dataset is a problem for you, you will probably have to move to suppress-all.
Recommendation: Use the Command Line Interface to Automate Testing
Intel Inspector XE must execute your code path to find errors in it. Thus, you should run the Intel Inspector XE on multiple versions of your code, on different workloads that stress different code paths, as well as on corner cases. The most effective way to do this is to run your application under the Intel Inspector XE multiple times with different data sets passed in as arguments. This suggests that batch processing using a CLI may simplify collecting all data.
Furthermore, given the inherent time dilation that comes with code-inspection tools, it is more efficient to run these tests overnight or as part of your regression testing suite and have the computer do the work for you; you just examine the results of multiple tests in the morning. Reports from the CLI can be exported in a number of parse-friendly formats, making it easier to generate automated mailings or write scripts to run on the resultant data.
To get help on inspxe-cl, use the -help CLI option and see more information in our installed and online help.
Quality assurance is a vital step in the production of any software used by external customers or internal customers. Intel Inspector XE has the right functionality and features to help you collect the QA correctness information you need. Using the Intel Inspector XE features correctly and choosing your data sets wisely helps you produce the best results. Taking advantage of suppression and filtering functionality can help you focus on addressing new issues, particularly in an automated testing environment.