Roofline Analysis Purpose and Usage
- What is the maximum achievable performance with your current hardware resources?
- Does your application work optimally on current hardware resources?
- If not, what are the best candidates for optimization?
- Is memory bandwidth or compute capacity limiting performance for each optimization candidate?
- Self data = Memory access, FLOPs, and duration related only to the loop/function itself and excludes data originating in other loops/functions called by it
- Total data = Data from the loop/function itself and its inner loops/functions
- Investigate the source of loops/functions instead of just the loops/functions themselves.
- Get a more accurate view of loops/functions that behave differently when called under different circumstances.
- Uncover design inefficiencies higher up the call chain that could be the root cause of poor performance by smaller loops/functions.
Run a Roofline Analysis
- Measure the hardware limitations of your machine and collect loop/function timings using the Survey analysis.
- Collect FLOP and integer operations data, and memory traffic data, using the Trip Counts and FLOP analysis - this collection can take three to four times longer than the Survey analysis.
- Run the Roofline analysis with theWith Callstackscheckbox enabled. Upon completion, theIntel Advisordisplays aRooflinechart.
- Enable theWith Callstackscheckbox in theRooflinechart.
- Run the Roofline analysis with theFor All Memory Levelscheckbox enabled. Upon completion, the Intel Advisor displays aRooflinechart.
- Afterwards, you can use the Roofline chart controls to select which memory levels to show dots for. For details, see theMemory LevelandShow memory level relationshipsoptions in the section below.
Roofline Chart Controls
Click to toggle between
Rooflinechart view and
Click to toggle to and from side-by-side
Drag to adjust the dimensions of the
Coresdrop-down toolbar to:
Choose the appropriate number of CPU cores to scale roof values up or down:
By default, the number of cores is set to the number of threads used by the application (even values only).
You’ll see the following options if your code is running on a multisocket PC:
Rooflinechart data from other
Intel Advisorresults or non-archived snapshots for comparison purposes.
Use the drop-down toolbar to:
Click a loop/function dot in the current result to show the relationship (arrowed lines) between it and the corresponding loop/function dots in loaded results/snapshots.
Add visual indicators to the Roofline chart to make the interpretation of data easier, including performance limits and whether loops/functions are memory bound, compute bound, or both.
Use the drop-down toolbar to:
The preview picture is updated as you select guidance options, allowing you to see how changes will affect the Roofline chart’s appearance. Click
Applyto apply your changes, or
Defaultto return the Roofline chart to its original appearance.
You can save your Roofs Settings or Point Weight Representation configuration to a JSON file or load a custom configuration.
Zoom in and out using numerical values.
Hover your mouse over an item to display metrics for it.
Click a loop/function dot to:
You can also click an item in the
Callstackpane to flash the corresponding loop/function dot in the
If Roofline with Callstacks is enabled, click a loop/function dot control to collapse descendant dots into the parent dot, or click a loop/function dot control to show descendant dots and their relationship via visual indicators to the parent dot.
Right-click a loop/function dot or a blank area in the
Rooflinechart to perform more functions, such as:
If For All Memory Levels is enabled, double-click a a loop/function dot to display labeled dots representing the loop/function at selected memory levels (L1, L2, L3, DRAM). For details, see the
Show memory level relationshipsoptions described above.
If Roofline with Callstacks is enabled, show/hide the
Display the number and percentage of loops in each loop weight representation category.
Roofline Chart Data
- Arithmetic intensity (x axis) - measured in number of floating-point operations (FLOPs) and/or integer operations (INTOPs) per byte, based on the loop/function algorithm, transferred between CPU/VPU and memory
- Performance (y axis) - measured in billions of floating-point operations per second (GFLOPS) and/or billions of integer operations per second (GINTOPS)
- The size and color of eachRooflinechart dot represent relative execution time for each loop/function. Large red dots take the most time, so are the best candidates for optimization. Small green dots take less time, so may not be worth optimizing.
- Rooflinechart diagonal lines indicate memory bandwidth limitations preventing loops/functions from achieving better performance without some form of optimization. For example: TheL1 Bandwidthroofline represents the maximum amount of work that can get done at a given arithmetic intensity if the loopalwayshits L1 cache. A loop does not benefit from L1 cache speed if a dataset causes it to miss L1 cache too often, and instead is subject to the limitations of the lower-speed L2 cache itishitting. So a dot representing a loop that misses L1 cache too often but hits L2 cache is positioned somewhere below theL2 Bandwidthroofline.
- Rooflinechart horizontal lines indicate compute capacity limitations preventing loops/functions from achieving better performance without some form of optimization. For example: TheScalar Add Peakrepresents the peak number of add instructions that can be performed by the scalar loop under these circumstances. TheVector Add Peakrepresents the peak number of add instructions that can be performed by the vectorized loop under these circumstances. So a dot representing a loop that is not vectorized is positioned somewhere below theScalar Add Peakroofline.
- A dot cannot exceed the topmost rooflines, as these represent the maximum capabilities of the machine; however, not all loops can utilize maximum machine capabilities.
- The greater the distance between a dot and the highest achievable roofline, the more opportunity exists for performance improvement.
- A navigable, color-codedCallstackpane that shows the entire call chain for the selected loop/function, but excludes its callees
- Visual indicators (caller and callee arrows) that show the relationship among loops and functions
- The ability to simplify dot-heavy charts by collapsing several small loops into one overall representationLoops/functions with no self data are grayed out when expanded and in color when collapsed. Loops/functions with self data display at the coordinates, size, and color appropriate to the data when expanded, but have a gray halo of the size associated with their total time. When such loops/functions are collapsed, they change to the size and color appropriate to their total time and, if applicable, move to reflect the total performance and total arithmetic intensity.
What Do I Do Next?
- Address memory bandwidth bottlenecks.
- Address compute capacity bottlenecks.
- Identify the real bottlenecks.