Troubleshooting issues with SEP in the Embedded Tool suite – Intel® System Studio

Background

The Sampling Enabling Product (SEP) is a powerful tool for collecting event based profiling data on a system. SEP uses a hardware based, low-overhead and system-wide collector that helps identify the modules and functions that are consuming the most time. This article discusses several techniques that we have found useful when troubleshooting issues with SEP.

Overview

In troubleshooting SEP issues on embedded Linux* systems, there are several steps you should take:

  1. Verify the sep drivers are loaded
  2. Verify you have enough disk space to run sep
  3. Verify/increase size of the tmp partition
  4. Verify/increase the stack size
  5. Disable nmi watchdog timer
  6. Allow kernel symbols to be visible to data collector

 

Loading the sep driver

Run the command:  “./insmod-sep3” .

  • Run the command: “lsmod | grep sep”. You should see the sep driver listed.
    • If you don’t see sep, then you should run “dmesg | grep sep”
      • this will give you the kernel messages that are associated with sep
  • ls -l /dev/sep3_8 /dev/pax
    • You should see these devices and also make sure that you have permissions to access them.

 

 

Increasing the size of the /tmp partition

  1. Open /etc/init.d/system_setup with your editor of choice
  2. Comment out mount_if_not_mounted “/tmp” “tmpfs” “tmpfs” “-o size=8m”

 

Configuring stack size

  • Increase the stack size:
    • Run the command ulimit –s
    • This will give you the stacksize currently configured for the system.
  • ulimit –s unlimit 
    • To increase the stack size to unlimited.
  • ulimit –s 2048
    • It is possible the unlimit will not work on some systems, in this case you may need to tailor the stack size to be the maximum allowable.
  • Verify that the stack size limit has increased from 1024 to 2048 by running the command “ulimit –s” again.

Disable nmi watchdog timer

The NMI Watchdog can be used in the Linux* kernel to periodically detect if the CPU is locked. When CPU-locking occurs, the NMI Watchdog service does the following: 

  • Print debug info
  • Reboot the system.

However NMI  Watchdog needs to use a PMU counter, so other performance tools including SEP can’t use PMU event-based sampling data collections.

To disable the nmi watchdog timer use the following command:

echo 0 > /proc/sys/kernel/nmi_watchdog

Allow kernel symbols to be visible to data collector

if “cat /proc/sys/kernel/kptr_restrict” is anything but 0, then

echo 0 > /proc/sys/kernel/kptr_restrict

This allows the kernel symbols to be visible to the data collector.  If not set, you will see all the kernel samples attributed to “Unknown module” or “Other64” or “Other32”.

 

有关编译器优化的更完整信息,请参阅优化通知