Intel® VTune™ Profiler Performance Analysis Cookbook

Cookbook

  • 2021
  • 08/24/2021
  • Public Content
Contents

Real-Time Monitoring with System Analyzer

This recipe introduces the System Analyzer and shows how to monitor a target system in real-time to identify whether it is CPU, GPU, memory, disk, or network bound.
Content expert:
Jeffrey K. Reinemann
System Analyzer enables continuous real-time monitoring and reporting of basic performance and resource utilization data. It is integrated into Intel® VTune™ Amplifier 2019 Update 2 as a PREVIEW feature. Based on user feedback, it may be enhanced to improve usability and user experience, or removed in a future update or release.
Intel® VTune™ Profiler is a new renamed version of the Intel® VTune™ Amplifier.
The
target system
 on which data is collected can be either a Linux* or Windows* system, and the
host system
 where the GUI displays selected data can be either Linux, macOS*, or Windows system. System Analyzer can monitor a target system in near real-time and, based on these high-level results, you can configure an appropriate VTune Amplifier analysis for deep-dive profiling.
System Analyzer consists of a target data collector and a graphical user interface (GUI) to display timeline graphs of the performance metrics. The following performance metrics and display options are available:
  • processor utilization and processor frequency
  • memory utilization (Linux only)
  • network throughput (Linux only)
  • GPU utilization and other graphics metrics (with Intel Graphics Driver installed)
System Analyzer is also a part of the Intel® Graphics Performance Analyzers (Intel GPA) package. To analyze graphics rendering workloads and bottlenecks, use the System Analyzer and other tools from the Intel GPA package.

Ingredients

This section lists the software tools used for the performance analysis scenario.
  • Tools
    : Intel VTune Amplifier 2019 with the System Analyzer package
    The installation folder of the System Analyzer package on the host is
    <
    install-dir
    >/system_analyzer
    , where
    <install-dir>
     depends on the version of the VTune Amplifier:
    • on Windows* and Linux* systems, standalone or installed as part of Intel® oneAPI Base Toolkit:
      • Windows OS:
        [Program Files]\Intel\oneAPI\vtune\<
        version
        >
      • Linux OS:
        /opt/intel/oneapi/vtune/<
        version
        >
    • on Apple* macOS* systems:
      /Applications/Intel VTune Profiler <
      version
      >.app
    If the target system to be profiled has already been profiled by the VTune Amplifier, VTune Amplifier files are in the directory configured from the host VTune Amplifier, for example:
    /opt/intel/vtune_amplifier_
    version
     (default for the root user) or
    /tmp/vtune_amplifier_
    version
     (default for a non-root user). The target System Analyzer files are in the
    system_analyzer/target/
     subdirectory.

Launch System Analyzer

Prerequisites
:
  • If your target system is different from the host system, copy the target data collector files to the target system.
  • To run
    gpa_router
     on a Linux target system as a non-root user, set
    /proc/sys/dev/i915/perf_stream_paranoid
     to 0 with: 
    echo 0 > /proc/sys/dev/i915/perf_stream_paranoid
  1. Set up the System Analyzer collector on the target system:
    • On the Windows target system, navigate to the System Analyzer collector directory and start
      gpa_router
      , optionally specifying the IPv4 address of the host system to run the System Analyzer GUI and allow it to connect to this instance. 
      target> gpa_router.exe --ip-whitelist 10.7.158.142
    • On the Linux target system, navigate to the System Analyzer target collector directory and start
      gpa_router
      , optionally specifying the IPv4 address of the host system to run the System Analyzer GUI and allow it to connect to this instance: 
      target> ./gpa_router --ip-whitelist 10.0.0.2
      When the System Analyzer GUI connects to the target system,
      gpa_router
       reports the incoming connection. For example: 
      target> ./gpa_router --ip-whitelist 10.0.0.2

Start listening for new connections from port #27072
New incoming connection established with 10.0.0.2.
    For
    gpa_router
     command line options, including the syntax for specifying TCP ports to use, enter
    gpa_router --help
    .
  2. Launch the System Analyzer.
    For example, on the Windows host system: 
    host> SystemAnalyzer.exe
    If the
    gpa_router
     running on the target system reports an error "
    Incoming connection from host <IPv4> was rejected since address is not in ip-whitelist.
    ", restart
    gpa_router
     with the appropriate host system's IPv4 address specified with the
    --ip-whitelist
     option.
  3. Specify
    localhost
     or the target system's name or IPv4 address and click the
    Connect
     button:

Set Up System View

  1. After the System Analyzer connects to the target system, select the
    System View
     option under
    System Profiling
    :
  2. In the
    System View
    , specify the metrics to monitor by using the mouse to drag a metric to the graph display area. Add additional counters by holding the Control (Ctrl) key while dragging them to the graph display area. Add additional timeline graphs by dragging a metric below the bottom timeline graph.
    For example, configure timeline graphs for CPU utilization and frequency, network I/O, and disk I/O:
To export the data currently selected for display in the graphs to a comma-delimited file, click the
CSV
 icon at the top of the System Analyzer GUI window. The System Analyzer GUI briefly displays the directory and the
*.csv
 filename receiving the exported data on the title bar.

Identify Anomalies for Deeper Analysis

If the System Analyzer shows semi-random, periodic anomalies (for example, your target system has an expected network load that occasionally and unexpectedly drops to near zero), you need to further deep-dive to investigate and root-cause with the VTune Amplifier Input and Output or Microarchitecture Exploration analyses.
In this example, the graph indicates that there are several times when disk writes and network traffic simultaneously dip slightly. However, there is one instance where disk writes spike to more than double the usual rate and network traffic almost simultaneously drops to near zero:
This indicates that an in-depth VTune Amplifier data collection should start just before disk writes spike and stop a few seconds after that. In this case, accurately predicting the disk write spikes ahead of time in this scenario is unlikely, so the data collection should be configured to use the ring buffer to only save the last few seconds of data surrounding the disk write spike and network drop.

Configure VTune Amplifier for Anomaly Detection

Now you can configure VTune Amplifier to profile the system deeper using the ring buffer where only the last one to twenty seconds of data is preserved (and written to disk).
In the VTune Amplifier GUI, configure the ring buffer using the
Limit collected data by: Time from collection end
 option in the Attach to Process, Profile System, or Launch Application target configuration on the
WHAT
 pane:
To configure the ring buffer and run the analysis from the command line, use the
--ring-buffer
 option.

Run VTune Amplifier with System Analyzer

Simultaneously run both the VTune Amplifier analysis for detailed performance metrics and System Analyzer to monitor metrics display and identify when to stop VTune Amplifier data collection. This helps you analyze the system in the seconds before, during, and immediately after the unexpected behavior, and also minimizes the collection overhead by reducing disk writes and storage requirements for long collections.
For disk I/O issues detected with the System Analyzer in this recipe, run the Input and Output Analysis in the
System Disk IO API
 mode (default).
To run the VTune Amplifier collection from the command line, enter: 
amplxe-cl -collect io -knob collect-memory-bandwidth=true -ring-buffer=20 --duration unlimited
View the collected data in the Input and Output viewpoint. You can further narrow the investigation by filtering in and zooming in to the part of the timeline surrounding the disk write spike and the network traffic drop. For example, in the Bottom-up window:
In the Platform window:

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.