Troubleshooting issues with SEP in the Embedded Tool suite – Intel® System Studio

By Kevin P O'Leary, Published: 11/30/2012, Last Updated: 11/30/2012

Background

The Sampling Enabling Product (SEP) is a powerful tool for collecting event based profiling data on a system. SEP uses a hardware based, low-overhead and system-wide collector that helps identify the modules and functions that are consuming the most time. This article discusses several techniques that we have found useful when troubleshooting issues with SEP.

Overview

In troubleshooting SEP issues on embedded Linux* systems, there are several steps you should take:

  1. Verify the sep drivers are loaded
  2. Verify you have enough disk space to run sep
  3. Verify/increase size of the tmp partition
  4. Verify/increase the stack size
  5. Disable nmi watchdog timer
  6. Allow kernel symbols to be visible to data collector

 

Loading the sep driver

Run the command:  “./insmod-sep3” .

  • Run the command: “lsmod | grep sep”. You should see the sep driver listed.
    • If you don’t see sep, then you should run “dmesg | grep sep”
      • this will give you the kernel messages that are associated with sep
  • ls -l /dev/sep3_8 /dev/pax
    • You should see these devices and also make sure that you have permissions to access them.

 

 

Increasing the size of the /tmp partition

  1. Open /etc/init.d/system_setup with your editor of choice
  2. Comment out mount_if_not_mounted “/tmp” “tmpfs” “tmpfs” “-o size=8m”

 

Configuring stack size

  • Increase the stack size:
    • Run the command ulimit –s
    • This will give you the stacksize currently configured for the system.
  • ulimit –s unlimit 
    • To increase the stack size to unlimited.
  • ulimit –s 2048
    • It is possible the unlimit will not work on some systems, in this case you may need to tailor the stack size to be the maximum allowable.
  • Verify that the stack size limit has increased from 1024 to 2048 by running the command “ulimit –s” again.

Disable nmi watchdog timer

The NMI Watchdog can be used in the Linux* kernel to periodically detect if the CPU is locked. When CPU-locking occurs, the NMI Watchdog service does the following: 

  • Print debug info
  • Reboot the system.

However NMI  Watchdog needs to use a PMU counter, so other performance tools including SEP can’t use PMU event-based sampling data collections.

To disable the nmi watchdog timer use the following command:

echo 0 > /proc/sys/kernel/nmi_watchdog

Allow kernel symbols to be visible to data collector

if “cat /proc/sys/kernel/kptr_restrict” is anything but 0, then

echo 0 > /proc/sys/kernel/kptr_restrict

This allows the kernel symbols to be visible to the data collector.  If not set, you will see all the kernel samples attributed to “Unknown module” or “Other64” or “Other32”.

 

Product and Performance Information

1

Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804