Real-Time Monitoring with System Analyzer
This recipe introduces the System Analyzer and shows how to monitor a target system in real-time to identify whether it is CPU, GPU, memory, disk, or network bound.
Content expert:
Jeffrey K. Reinemann System Analyzer enables continuous real-time monitoring and reporting of basic performance and resource utilization data. It is integrated into Intel® VTune™ Amplifier 2019 Update 2 as a PREVIEW feature. Based on user feedback, it may be enhanced to improve usability and user experience, or removed in a future update or release.
Intel® VTune™ Profiler is a new renamed version of the Intel® VTune™ Amplifier.
The
target system
on which data is collected can be either a Linux* or Windows* system, and the
host system
where the GUI displays selected data can be either Linux, macOS*, or Windows system. System Analyzer can monitor a target system in near real-time and, based on these high-level results, you can configure an appropriate VTune Amplifier analysis for deep-dive profiling.
System Analyzer consists of a target data collector and a graphical user interface (GUI) to display timeline graphs of the performance metrics. The following performance metrics and display options are available:
- processor utilization and processor frequency
- memory utilization (Linux only)
- network throughput (Linux only)
- GPU utilization and other graphics metrics (with Intel Graphics Driver installed)
System Analyzer is also a part of the Intel® Graphics Performance Analyzers (Intel GPA) package. To analyze graphics rendering workloads and bottlenecks, use the System Analyzer and other tools from the Intel GPA package.
Ingredients
This section lists the software tools used for the performance analysis scenario.
- Tools: Intel VTune Amplifier 2019 with the System Analyzer packageThe installation folder of the System Analyzer package on the host is<, whereinstall-dir>/system_analyzer<install-dir>depends on the version of the VTune Amplifier:
- on Windows* and Linux* systems, standalone or installed as part of Intel® oneAPI Base Toolkit:
- Windows OS:[Program Files]\Intel\oneAPI\vtune\<version>
- Linux OS:/opt/intel/oneapi/vtune/<version>
- on Apple* macOS* systems:/Applications/Intel VTune Profiler <version>.app
If the target system to be profiled has already been profiled by the VTune Amplifier, VTune Amplifier files are in the directory configured from the host VTune Amplifier, for example:/opt/intel/vtune_amplifier_(default for the root user) orversion/tmp/vtune_amplifier_(default for a non-root user). The target System Analyzer files are in theversionsystem_analyzer/target/subdirectory.- In all cases, the target system collector files are in thetargetsub-directory and the host GUI is in thehostsub-directory.
- For the list of host and target operating systems supported by the System Analyzer, see Supported Platforms and Applications.
- For trial VTune Amplifier downloads and product support, visit https://software.intel.com/en-us/vtune.
Launch System Analyzer
Prerequisites
:
- If your target system is different from the host system, copy the target data collector files to the target system.
- To rungpa_routeron a Linux target system as a non-root user, set/proc/sys/dev/i915/perf_stream_paranoidto 0 with:echo 0 > /proc/sys/dev/i915/perf_stream_paranoid
- Set up the System Analyzer collector on the target system:
- On the Windows target system, navigate to the System Analyzer collector directory and startgpa_router, optionally specifying the IPv4 address of the host system to run the System Analyzer GUI and allow it to connect to this instance.target> gpa_router.exe --ip-whitelist 10.7.158.142
- On the Linux target system, navigate to the System Analyzer target collector directory and startgpa_router, optionally specifying the IPv4 address of the host system to run the System Analyzer GUI and allow it to connect to this instance:target> ./gpa_router --ip-whitelist 10.0.0.2When the System Analyzer GUI connects to the target system,gpa_routerreports the incoming connection. For example:target> ./gpa_router --ip-whitelist 10.0.0.2 Start listening for new connections from port #27072 New incoming connection established with 10.0.0.2.
Forgpa_routercommand line options, including the syntax for specifying TCP ports to use, entergpa_router --help. - Launch the System Analyzer.For example, on the Windows host system:host> SystemAnalyzer.exeIf thegpa_routerrunning on the target system reports an error "Incoming connection from host <IPv4> was rejected since address is not in ip-whitelist.", restartgpa_routerwith the appropriate host system's IPv4 address specified with the--ip-whitelistoption.
- Specifylocalhostor the target system's name or IPv4 address and click theConnectbutton:
Set Up System View
- After the System Analyzer connects to the target system, select theSystem Viewoption underSystem Profiling:
- In theSystem View, specify the metrics to monitor by using the mouse to drag a metric to the graph display area. Add additional counters by holding the Control (Ctrl) key while dragging them to the graph display area. Add additional timeline graphs by dragging a metric below the bottom timeline graph.For example, configure timeline graphs for CPU utilization and frequency, network I/O, and disk I/O:
To export the data currently selected for display in the graphs to a comma-delimited file, click the
CSV
icon at the top of the System Analyzer GUI window. The System Analyzer GUI briefly displays the directory and the
*.csv
filename receiving the exported data on the title bar.
Identify Anomalies for Deeper Analysis
If the System Analyzer shows semi-random, periodic anomalies (for example, your target system has an expected network load that occasionally and unexpectedly drops to near zero), you need to further deep-dive to investigate and root-cause with the VTune Amplifier Input and Output or Microarchitecture Exploration analyses.
In this example, the graph indicates that there are several times when disk writes and network traffic simultaneously dip slightly. However, there is one instance where disk writes spike to more than double the usual rate and network traffic almost simultaneously drops to near zero:

This indicates that an in-depth VTune Amplifier data collection should start just before disk writes spike and stop a few seconds after that. In this case, accurately predicting the disk write spikes ahead of time in this scenario is unlikely, so the data collection should be configured to use the ring buffer to only save the last few seconds of data surrounding the disk write spike and network drop.
Configure VTune Amplifier for Anomaly Detection
Now you can configure VTune Amplifier to profile the system deeper using the ring buffer where only the last one to twenty seconds of data is preserved (and written to disk).
In the VTune Amplifier GUI, configure the ring buffer using the
Limit collected data by: Time from collection end
option in the Attach to Process, Profile System, or Launch Application target configuration on the
WHAT
pane:

To configure the ring buffer and run the analysis from the command line, use the
--ring-buffer
option.
Run VTune Amplifier with System Analyzer
Simultaneously run both the VTune Amplifier analysis for detailed performance metrics and System Analyzer to monitor metrics display and identify when to stop VTune Amplifier data collection. This helps you analyze the system in the seconds before, during, and immediately after the unexpected behavior, and also minimizes the collection overhead by reducing disk writes and storage requirements for long collections.
For disk I/O issues detected with the System Analyzer in this recipe, run the Input and Output Analysis in the
System Disk IO API
mode (default).
To run the VTune Amplifier collection from the command line, enter:
amplxe-cl -collect io -knob collect-memory-bandwidth=true -ring-buffer=20 --duration unlimited
View the collected data in the Input and Output viewpoint. You can further narrow the investigation by filtering in and zooming in to the part of the timeline surrounding the disk write spike and the network traffic drop. For example, in the Bottom-up window:

In the Platform window:
