Test Prioritization Tool

Intel® Fortran Compiler Classic and Intel® Fortran Compiler Developer Guide and Reference

Download PDF

ID 767251

Date 9/08/2022

Version

Public

A newer version of this document is available. Customers should click here to go to the newest version.

Visible to Intel only — GUID: GUID-8DCD784F-4BF5-4DDE-9105-CF34A6639239

View Details

Test Prioritization Tool

The test prioritization tool, also known as the tselect tool, enables the profile-guided optimizations on all supported Intel® architectures, on Linux*, Windows*, and macOS operating systems, to select and prioritize tests for an application based on prior execution profiles.

The tool offers a potential of significant time saving in testing and developing large-scale applications where testing is the major bottleneck.

Development often requires changing applications modules. As applications change, developers can have a difficult time retaining the quality of their functional and performance tests so they are current and on-target. The test prioritization tool lets software developers select and prioritize application tests as application profiles change.

The information about the tool is separated into the following sections:

Features and benefits
Requirements and syntax
Usage model
Tool options
Running the tool

Features and Benefits

The test prioritization tool provides an effective testing hierarchy based on the code coverage for an application. The following list summarizes the advantages of using the tool:

Minimizing the number of tests that are required to achieve a given overall coverage for any subset of the application. The tool defines the smallest subset of the application tests that achieve exactly the same code coverage as the entire set of tests.
Reducing the turn-around time of testing. Instead of spending a long time on finding a possibly large number of failures, the tool enables the users to quickly find a small number of tests that expose the defects associated with the regressions caused by a change set.
Selecting and prioritizing the tests to achieve certain level of code coverage in a minimal time based on the data of the tests' execution time.

See Understanding Profile-guided Optimization and Profile an Application topics for general information on creating the files needed to run this tool.

Test Prioritization Tool Requirements

The test prioritization tool needs the following items to work:

The .spi file generated by Intel® compilers when compiling the application for the instrumented binaries with the -prof-gen=srcpos (Linux* and macOS) or /Qprof-gen:srcpos (Windows*) option.
The .dpi files generated by the profmerge tool as a result of merging the dynamic profile information .dyn files of each of the application tests. Run the profmerge tool on all .dyn files that are generated for each individual test and name the resulting .dpi in a fashion that uniquely identifies the test.

User-generated file containing the list of tests to be prioritized. For successful instrumented code run, you should:

Name each test .dpi file so the file names uniquely identify each test.
Create a .dpi list file, which is a text file that contains the names of all .dpi test files.

Each line of the .dpi list file should include one, and only one .dpi file name. The name can optionally be followed by the duration of the execution time for a corresponding test in the dd:hh:mm:ss format.

For example: Test1.dpi 00:00:60:35 states that Test1 lasted 0 days, 0 hours, 60 minutes and 35 seconds.

The execution time is optional. However, if it is not provided, then the tool will not prioritize the test for minimizing execution time. It will prioritize to minimize the number of tests only.

CAUTION:

The profmerge tool merges all .dyn files that exist in the given directory. Make sure unrelated .dyn files, which may remain from unrelated runs, are not present. Otherwise, the profile information will be skewed with invalid profile data, which can result in misleading coverage information and adverse performance of the optimized code. The tool uses the following general syntax:

Tool Syntax
`tselect -dpi_listfile`

-dpi_list is a required tool option that sets the path to the list file containing the list of the all .dpi files. All other tool commands are optional.

NOTE:

Windows* only: Unlike the compiler options, which are preceded by forward slash ("/"), the tool options are preceded by a hyphen ("-").

Usage Model

The following figure illustrates a typical test prioritization tool usage model.

Test Prioritization Tool Options

The tool uses the options that are listed in the following table:

Option	Description
-help	Prints tool option descriptions.
-dpi_list`file`	Required. Specifies the name of the file that contains the names of the dynamic profile information (.dpi) files. Each line of the file must contain only one .dpi file name, which can be followed by its execution time (optional). The name must uniquely identify the test.
-spi`file`	Specifies the file name of the static profile information file (.SPI). Default is pgopti.spi
-o`file`	Specifies the file name of the output report file.
-comp`file`	Specifies the file name that contains the list of files of interest.
-cutoff`value`	Instructs the tool to terminate when the cumulative block coverage reaches a preset percentage, as specified by `value`, of pre-computed total coverage. `value` must be greater than 0.0 (for example, 99.00) but not greater than 100. `value` can be set to 100.
-nototal	Instructs the tool to ignore the pre-compute total coverage process.
-mintime	Instructs the tool to minimize testing execution time. The execution time of each test must be provided on the same line of dpi_list file, after the test name in dd:hh:mm:ss format.
-srcbasedir`dir`	Specifies a different top-level project directory than was used during compiler instrumentation run with the prof-src-root compiler option to support relative paths to source files in place of absolute paths.
-verbose	Instructs the tool to generate more logging information about program progress.

Running the Tool

The following steps demonstrate one simple example for running the tool on IA-32 architectures.

Specify the directory by entering a command similar to the following:

Example

set PROF_DIR=c:\myApp\prof-dir

Example
`set PROF_DIR=c:\myApp\prof-dir`

Compile the program and generate instrumented binary by issuing commands similar to the following:

Operating System	Command
Linux and macOS	`ifort -prof-gen=srcpos myApp.f90`
Windows	`ifort /Qprof-gen:srcpos myApp.f90`

The commands shown above compile the program and generate instrumented binary myApp, as well as the corresponding static profile information pgopti.spi.

Confirm that unrelated .dyn files are not present by issuing a command similar to the following:

Example

rm prof-dir \*.dyn
Run the instrumented files by issuing a command similar to the following:

Example

myApp < data1

The command runs the instrumented application and generates one or more new dynamic profile information files that have an extension .dyn in the directory specified by the -prof-dir step above.
Merge all .dyn file into a single file by issuing a command similar to the following:

Example

profmerge -prof_dpi Test1.dpi

The profmerge tool merges all the .dyn files into one file (Test1.dpi) that represents the total profile information of the application on Test1.
Confirm again there are no unrelated .dyn files present a second time by issuing a command similar to the following:

Example

rm prof-dir \*.dyn
Run the instrumented application, and generate one or more new dynamic profile information files that have an extension .dyn in the directory specified in the prof-dir step above by issuing a command similar to the following:

Example

myApp < data2
Merge all .dyn files into a single file by issuing a command similar to the following:

Example

profmerge -prof_dpi Test2.dpi

At this step, the profmerge tool merges all the .dyn files into one file (Test2.dpi) that represents the total profile information of the application on Test2.
Confirm that there are no unrelated .dyn files present for the final time by issuing a command similar to the following:

Example

rm prof-dir \*.dyn
Run the instrumented application and generate one or more new dynamic profile information files that have an extension .dyn in the directory specified by -prof-dir by issuing a command similar to the following:

Example

myApp < data3
Merge all .dyn file into a single file, by issuing a command similar to the following:

Example

profmerge -prof_dpi Test3.dpi

At this step, the profmerge tool merges all the .dyn files into one file (Test3.dpi) that represents the total profile information of the application on Test3.
Create a file named tests_list with three lines. The first line contains Test1.dpi, the second line contains Test2.dpi, and the third line contains Test3.dpi.

Example
`rm prof-dir \*.dyn`

Example
`myApp < data1`

Example
`profmerge -prof_dpi Test1.dpi`

Example
`rm prof-dir \*.dyn`

Example
`myApp < data2`

Example
`profmerge -prof_dpi Test2.dpi`

Example
`rm prof-dir \*.dyn`

Example
`myApp < data3`

Example
`profmerge -prof_dpi Test3.dpi`

Tool Usage Examples

When these items are available, the test prioritization tool may be launched from the command line in the prof-dir directory as described in the following examples.

Example 1: Minimizing the Number of Tests

The following example describes how minimize the number of test runs.

Example Syntax
`tselect -dpi_list tests_list -spi pgopti.spi`

where the -spi option specifies the path to the .spi file.

The following sample output shows typical results.

Sample Output
Total number of tests = 3 Total block coverage ~ 52.17 Total function coverage ~ 50.00 num %RatCvrg %BlkCvrg %FncCvrg Test Name @ Options --- -------- -------- -------- ------------------- 1 87.50 45.65 37.50 Test3.dpi 2 100.50 52.17 50.00 Test2.dpi

Sample Output

Total number of tests   =  3

Total block coverage    ~  52.17

Total function coverage ~  50.00

 num  %RatCvrg  %BlkCvrg  %FncCvrg  Test Name @ Options

 ---  --------  --------  --------  -------------------

   1   87.50     45.65     37.50    Test3.dpi

   2   100.50    52.17     50.00    Test2.dpi

In this example, the results provide the following information:

By running all three tests, you achieve 52.17% block coverage and 50.00% function coverage.
Test3 alonecovers 45.65% of the basic blocks of the application, which is 87.50% of the total block coverage that can be achieved from all three tests.
By adding Test2, you achieve a cumulative block coverage of 52.17% or 100% of the total block coverage of Test1, Test2, and Test3.
Elimination of Test1 has no negative impact on the total block coverage.

Example 2: Minimizing Execution Time

Assume you have the following execution time of each test in the tests_list file:

Sample Output
Test1.dpi 00:00:60:35 Test2.dpi 00:00:10:15 Test3.dpi 00:00:30:45

Sample Output

Test1.dpi 00:00:60:35

Test2.dpi 00:00:10:15

Test3.dpi 00:00:30:45

The following command minimizes the execution time by passing the -mintime option:

Sample Syntax
`tselect -dpi_list tests_list -spi pgopti.spi -mintime`

The following sample output shows possible results:

Sample Output
Total number of tests = 3 Total block coverage ~ 52.17 Total function coverage ~ 50.00 Total execution time = 1:41:35 num elaspedTime %RatCvrg %BlkCvrg %FncCvrg Test Name @ Options --- ----------- -------- -------- -------- ------------------- 1 10:15 75.00 39.13 25.00 Test2.dpi 2 41:00 100.00 52.17 50.00 Test3.dpi

Sample Output

Total number of tests   =  3

Total block coverage    ~  52.17

Total function coverage ~  50.00

Total execution time    =  1:41:35

 num  elaspedTime  %RatCvrg  %BlkCvrg  %FncCvrg  Test Name @ Options

 ---  -----------  --------  --------  --------  -------------------

   1   10:15        75.00     39.13     25.00    Test2.dpi

   2   41:00        100.00    52.17     50.00    Test3.dpi

In this case, the results indicate that running all tests sequentially would require one hour, 45 minutes, and 35 seconds, while the selected tests would achieve the same total block coverage in only 41 minutes.

The order of tests when based on minimizing time (first Test2, then Test3) may be different than when prioritization is done based on minimizing the number of tests. See Example 1 shown above: first Test3, then Test2. In Example 2, Test2 is the test that gives the highest coverage per execution time, so Test2 is picked as the first test to run.

Using Other Options

The -cutoff enables the tool to exit when it reaches a given level of basic block coverage. The following example demonstrates how to use the option:

Example
`tselect -dpi_list tests_list -spi pgopti.spi -cutoff 85.00`

If the tool is run with the cutoff value of 85.00, as in the above example, only Test3 will be selected, as it achieves 45.65% block coverage, which corresponds to 87.50% of the total block coverage that is reached from all three tests.

The tool does an initial merging of all the profile information to figure out the total coverage that is obtained by running all the tests. The -cutoff enables you to skip this step. In such a case, only the absolute coverage information will be reported, as the overall coverage remains unknown.

Parent topic: PGO Tools

Select Your Language

Using Intel.com Search

Quick Links

Recent Searches

Advanced Search

Only search in

Intel® Fortran Compiler Classic and Intel® Fortran Compiler Developer Guide and Reference

Test Prioritization Tool

Features and Benefits

Test Prioritization Tool Requirements

Usage Model

Test Prioritization Tool Options

Running the Tool

Example 1: Minimizing the Number of Tests

Example 2: Minimizing Execution Time

Using Other Options