This recipe introduces the System Analyzer and shows how to monitor a target system in real-time to identify whether it is CPU, GPU, memory, disk, or network bound.
Content expert: Jeffrey K. Reinemann
System Analyzer enables continuous real-time monitoring and reporting of basic performance and resource utilization data. It is integrated into Intel® VTune™ Amplifier 2019 Update 2 as a PREVIEW feature. Based on user feedback, it may be enhanced to improve usability and user experience, or removed in a future update or release.
The target system on which data is collected can be either a Linux* or Windows* system, and the host system where the GUI displays selected data can be either Linux, macOS*, or Windows system. System Analyzer can monitor a target system in near real-time and, based on these high-level results, you can configure an appropriate VTune Amplifier analysis for deep-dive profiling.
System Analyzer consists of a target data collector and a graphical user interface (GUI) to display timeline graphs of the performance metrics. The following performance metrics and display options are available:
processor utilization and processor frequency
memory utilization (Linux only)
network throughput (Linux only)
GPU utilization and other graphics metrics (with Intel Graphics Driver installed)
System Analyzer is also a part of the Intel® Graphics Performance Analyzers (Intel GPA) package. To analyze graphics rendering workloads and bottlenecks, use the System Analyzer and other tools from the Intel GPA package.
This section lists the software tools used for the performance analysis scenario.
Tools: Intel VTune Amplifier 2019 Update 2 with the System Analyzer package
The installation folder of the System Analyzer package on the host depends on the version of the VTune Amplifier:
For the standalone version with Linux host - Linux/Android target: /opt/intel/vtune_amplifier_version/system_analyzer
For the standalone version with Windows host - Linux/Android target: C:\[Program Files]\IntelSWTools\VTune Amplifierversion\system_analyzer
For the Intel System Studio version with Linux host - Linux/Android target: /opt/intel/system_studio_version/vtune_amplifier_version/system_analyzer
For the Intel System Studio version with Windows host - Linux/Android target: : C:\[Program Files]\IntelSWTools\system_studio_version\vtune_amplifier_version\system_analyzer
If the target system to be profiled has already been profiled by the VTune Amplifier, VTune Amplifier files are in the directory configured from the host VTune Amplifier, for example: /opt/intel/vtune_amplifier_version (default for the root user) or /tmp/vtune_amplifier_version (default for a non-root user). The target System Analyzer files are in the system_analyzer/target/ subdirectory.
In all cases, the target system collector files are in the target sub-directory and the host GUI is in the host sub-directory.
For the list of host and target operating systems supported by the System Analyzer, see Supported Platforms and Applications.
For trial VTune Amplifier downloads and product support, visit https://software.intel.com/en-us/vtune.
Launch System Analyzer
- If your target system is different from the host system, copy the target data collector files to the target system.
To run gpa_router on a Linux target system as a non-root user, set /proc/sys/dev/i915/perf_stream_paranoid to 0 with:
echo 0 > /proc/sys/dev/i915/perf_stream_paranoid
Set up the System Analyzer collector on the target system:
On the Windows target system, navigate to the System Analyzer collector directory and start gpa_router, optionally specifying the IPv4 address of the host system to run the System Analyzer GUI and allow it to connect to this instance.
target> gpa_router.exe --ip-whitelist 10.7.158.142
On the Linux target system, navigate to the System Analyzer target collector directory and start gpa_router, optionally specifying the IPv4 address of the host system to run the System Analyzer GUI and allow it to connect to this instance:
target> ./gpa_router --ip-whitelist 10.0.0.2
When the System Analyzer GUI connects to the target system, gpa_router reports the incoming connection. For example:
target> ./gpa_router --ip-whitelist 10.0.0.2 Start listening for new connections from port #27072 New incoming connection established with 10.0.0.2.
For gpa_router command line options, including the syntax for specifying TCP ports to use, enter gpa_router --help.
Launch the System Analyzer.
For example, on the Windows host system:
If the gpa_router running on the target system reports an error "Incoming connection from host <IPv4> was rejected since address is not in ip-whitelist.", restart gpa_router with the appropriate host system's IPv4 address specified with the --ip-whitelist option.
Specify localhost or the target system's name or IPv4 address and click the Connect button:
Set Up System View
After the System Analyzer connects to the target system, select the System View option under System Profiling:
In the System View, specify the metrics to monitor by using the mouse to drag a metric to the graph display area. Add additional counters by holding the Control (Ctrl) key while dragging them to the graph display area. Add additional timeline graphs by dragging a metric below the bottom timeline graph.
For example, configure timeline graphs for CPU utilization and frequency, network I/O, and disk I/O:
To export the data currently selected for display in the graphs to a comma-delimited file, click the CSV icon at the top of the System Analyzer GUI window. The System Analyzer GUI briefly displays the directory and the *.csv filename receiving the exported data on the title bar.
Identify Anomalies for Deeper Analysis
If the System Analyzer shows semi-random, periodic anomalies (for example, your target system has an expected network load that occasionally and unexpectedly drops to near zero), you need to further deep-dive to investigate and root-cause with the VTune Amplifier Input and Output or Microarchitecture Exploration analyses.
In this example, the graph indicates that there are several times when disk writes and network traffic simultaneously dip slightly. However, there is one instance where disk writes spike to more than double the usual rate and network traffic almost simultaneously drops to near zero:
This indicates that an in-depth VTune Amplifier data collection should start just before disk writes spike and stop a few seconds after that. In this case, accurately predicting the disk write spikes ahead of time in this scenario is unlikely, so the data collection should be configured to use the ring buffer to only save the last few seconds of data surrounding the disk write spike and network drop.
Configure VTune Amplifier for Anomaly Detection
Now you can configure VTune Amplifier to profile the system deeper using the ring buffer where only the last one to twenty seconds of data is preserved (and written to disk).
In the VTune Amplifier GUI, configure the ring buffer using the Limit collected data by: Time from collection end option in the Attach to Process, Profile System, or Launch Application target configuration on the WHAT pane:
To configure the ring buffer and run the analysis from the command line, use the --ring-buffer option.
Run VTune Amplifier with System Analyzer
Simultaneously run both the VTune Amplifier analysis for detailed performance metrics and System Analyzer to monitor metrics display and identify when to stop VTune Amplifier data collection. This helps you analyze the system in the seconds before, during, and immediately after the unexpected behavior, and also minimizes the collection overhead by reducing disk writes and storage requirements for long collections.
For disk I/O issues detected with the System Analyzer in this recipe, run the Input and Output Analysis in the System Disk IO API mode (default).
To run the VTune Amplifier collection from the command line, enter:
amplxe-cl -collect io -knob collect-memory-bandwidth=true -ring-buffer=20 --duration unlimited
View the collected data in the Input and Output viewpoint. You can further narrow the investigation by filtering in and zooming in to the part of the timeline surrounding the disk write spike and the network traffic drop. For example, in the Bottom-up window:
In the Platform window: