Intel Software Network articles Feed

Custom Performance Counters, Visual Studio*, and the VTune Performance Analyzer

Sun, 15 Aug 2010 21:00:00 -0700

Problem :
If you have a Visual Studio solution where one of the projects is a performance counter DLL, after installing the VTune analyzer and integrating it into Visual Studio, you may find that any attempt to build the DLL fails.

Environment :
Windows*

Root Cause :
The VTune Performance Analyzer, upon start up, launches a helper process, 'vtunecca.exe'. This helper process collects the list of performance objects and counters available in the system for configuration in preparation for Counter Monitor data collection. This procedure of loading the performance objects and counters causes all custom performance counter DLLs to be loaded and locked into that process. Consequently, when an attempt is made to build a custom performance counter DLL, the build fails because the DLL is open for access and cannot be overwritten.

Resolution :
There are a couple of workarounds for this known issue:

Un-integrate the VTune analyzer from Visual Studio. Since the VTune analyzer has a standalone graphic interface, you can un-integrate from Visual Studio and use the standalone GUI to collect data.
To un-integrate: in the Control Panel, go to Add/Remove Programs and select the VTune Performance Analyzer and then press the Change button. When the dialog opens, select Modify and step through the dialogs pressing Next until you get to the dialog that allows you to uncheck the integrate with Visual Studio option.
Rename the vtunecca.exe file prior to starting Visual Studio. You can find the vtunecca.exe file in the C:\Program Files\Intel\VTune\Analyzer\Bin directory. Of course, you cannot use the Counter Monitor while the name of this file is changed, but you will be able to build the DLL and test it using other means.
Simply rename the file back to the original vtunecca.exe and the Counter Monitor feature will begin working again, after restarting the VTune analyzer.

Call graph and Time Based Sampling problems on Microsoft Windows Server 2008 R2

Tue, 04 May 2010 21:00:00 -0700

Problem:
Time Based Sampling gives an error: "Failed to create sampling database. Probably .tb5 files are corrupted or don't exist." Call graph also produces an error: “ Error in Module.. Copying Module failed”

Environment:
Microsoft Windows* Server 2008 R2 and VTune(TM) analyzer for Windows Update 7

Root Cause:
Microsoft Windows* Server 2008 R2 is Windows* 7-based. This update (Update 7) of VTune(TM) analyzer supports Windows 7* OS with event based sampling. It has no support for Time Based Sampling or call graph.

Resolution:
For Time Based Sampling or call graph capabilities on Windows 7*, you can use Intel(R) Parallel Amplifier's "hotspot analysis" (statistical call tree). Parallel Amplifier can be downloaded from the Intel website (http://software.intel.com/en-us/intel-parallel-amplifier/ ) and will recognize a current, valid VTune analyzer license. We do plan to move the “hotspot” analysis into a future version of VTune analyzer, until then, please use Parallel Amplifier. If you need to use the existing call graph capability in VTune analyzer, please use it on Windows XP* or Vista*. For Time Based Sampling, you can use Event Based Sampling and select Clockticks.

Application Crashes When Attempting Call Graph Profiling

Fri, 30 Oct 2009 00:00:00 -0700

Application Crashes or No Data Collected When Attempting Call Graph Profiling

We have seen cases where a user's application will crash when the VTune analyzer launches the application for call graph profiling. Another symptom of this problem is the No results were generated message from the VTune analyzer, after the application under test completes. Because of our binary instrumentation technology, and the injecting of code into a user application, sometimes various Microsoft* runtime libraries will cause the application to crash. Symptoms vary, but you may see the "unhandled exception" message and the "Just-In-Time Debugger" prompt from Visual Studio*.

Followed by the VTune analyzer "No results were generated for this run" message:

Reducing the instrumentation level of the Microsoft runtime libraries will often enable you to workaround this problem. Follow these steps to apply the workaround:

1) Right-click on the Activity in the Tuning Browser and select Modify Collectors...:

2) After the Configure Call Graph dialog appears, scroll the list of modules until you locate the msvcrnn[d].dll and msvcrt.dll modules. The nn represents the version of Visual Studio that you are using. In our example, Visual Studio 2005 was used and, therefore, nn = "80", as in 8.0. If you are using Visual Studio 2008, the module would be msvcr90.dll or msvcr90d.dll, where the 'd' representing the debug version of the DLL.

Note that color-coding is used in the module list. An explanation of the colors follows, and is available in the online help, as well.

Gray: Modules you added to the project from the Application/Module Profile Configuration dialog box, or from the Call Graph Configuration Wizard.
Blue: Modules added during call graph instrumentation as dependencies of the selected modules.
White: Modules you added via Add button, or added during run-time instrumentation.

3)      Click on the Instrumentation Level cell for the msvcr80.dll, for example, and select Minimal:

4)      Press Apply and then Instrument Now:

5)      Now OK out of all dialogs and re-run the activity.

In this example, simply reducing the instrumentation level of msvcr80.dll resolved the problem and results were successfully collected. The effect of reducing the instrumentation level on the runtime libraries is that no information regarding calls into those modules will be available in the call graph data. If this information is critical for your tuning activity, you can try to use the Custom Instrumentation (see Functions... button) to deselect functions until the call graph activity succeeds. Using a binary search technique, you would narrow down which function or functions fail when they are instrumented and select to not instrument them. Follow these steps to accomplish this task:

1)      Press Functions... for the module of concern. You will see a dialog box such as this:

2)      Now, using the scroll bar move to the middle of the list and select that line:

3)      Press Shift-Ctrl-End to select from this line to the bottom of the list. The horizontal scroll bar will move all the way to the right. Simply grab it and move it back to the left:

4)      Press the Uncheck button:

5)      Press OK and then Apply and Instrument Now:

6)      Notice that Instrumentation Level now shows Custom for the selected module.

7)      OK out of all dialogs and re-run the Activity.

If the application fails, again, modify the function selection so that the top half of the functions is selected and re-run the activity. In this way, you can narrow down which function(s) do not like to be instrumented (or "fail" when instrumented).

An alternative to the VTune analyzer's Call Graph feature is the Intel® Parallel Amplifier's Hotspot analysis. Parallel Amplifier uses a new technology to collect periodic samples, similar to "Clockticks" in the VTune analyzer, with complete call stacks, so that the call path to the hotspot is visible without instrumentation. If you have a valid license for the VTune analyzer that includes support through April 1, 2010, you can download, install, and use Parallel Amplifier without any additional cost.

How Do I Measure Memory Bandwidth on an Intel® Core™ i7 or Xeon® 5500 Series Platform Using Intel® VTune™ Performance Analyzer?

Tue, 22 Sep 2009 00:00:00 -0700

Updated! 2/3/2011

The new Intel® Core™ i7 and Xeon® 5500 series processors have a different architecture than previous processors, particularly when it comes to the uncore. The “uncore” is the part of the processor that is external to all the individual cores- for example, on the Core™ i7, there are 4 cores, and these share an L3 cache and a memory interface. The L3 and memory interface are considered uncore. VTune™ Performance Analyzer does not support the sampling of events that are triggered in the uncore of the processor.

However, due to popular demand, we have created and documented a way for VTune analyzer users who have Core i7 or Xeon 5500 series processor-based platforms to measure memory bandwidth. This is not measurable by default since the events needed are in the uncore. Here is the process to enable bandwidth measurement using the program sep. Sep is a utility that provides the sampling functionality used by VTune analyzer and Intel® Performance Tuning Utility (PTU).

Note that with this method the bandwidth events are counted using time-based sampling, not the event-based sampling that VTune analyzer normally uses. This means that you can determine a bandwidth for your whole system over a designated time range, but you won’t be able to see how much of the bandwidth used came from various functions/processes/modules. You can only see the total bandwidth for the system. Please adjust your application testing accordingly by running only the target application while measuring bandwidth.

1. Download the Intel® Performance Tuning Utility 3.2 update 1. The version of sep needed for this method is only available in this release of PTU. If you have a 32-bit operating system, get the IA-32 version, and if you have a 64-bit operating system, get the Intel® 64 version. PTU is available for both Windows* and Linux*.
2. Un-compress the package and follow the instructions in INSTALL.txt to install PTU. Make sure to install the sampling driver!
3. Download the appropriate Uncore Measurement package and uncompress it into a directory of your choice. To download the package, to go http://premier.intel.com, log in, and select File Downloads from the menu on the left. Select either VTune™ Performance Analyzer for Linux* or VTune™ Performance Analyzer for Windows* and click Display File List. The package will be named lin_measurebw.tar.gz for Linux* or win_measurebw.zip for Windows*.
4. Run the bandwidth measurement script (uncore.bat for Windows*, uncore.sh for Linux*) from the uncore directory. This script sets up the environment needed to measure bandwidth, and then uses sep to measure it. It is important that you measure bandwidth using this script to avoid unstable configuration changes to your VTune™ analyzer or PTU installations! If you run this script from a command prompt (instead of double-clicking, close the command window afterwards.
5. Once the bandwidth measurement script has finished executing, open the bandwidth.txt file in the same directory. This file contains the results of bandwidth measurement, and will be overwritten each time you run the bandwidth measurement script. See the Interpreting Bandwidth section below to analyze the data.
6. Now that PTU is installed, you may use it for your sampling needs, or you can use VTune analyzer. PTU will be the current active sampling technology on your system after executing these instructions. You will need to follow these instructions for switching between using PTU and VTune analyzer for sampling.

Interpreting Bandwidth

This method measures bandwidth from each processor’s uncore memory controller to memory. It will include memory reads, memory writes, I/O, and writebacks from L3 to memory. It does not include traffic from cache-to-cache transfers between sockets.

Using this method, your bandwidth.txt results file will contain results in this format:

Version Info: Sampling Enabling Product version: 2.9.devbuild (private) built on Mar 18 2009 02:53:25 P:Intel(R) Xeon(R) Processor 5500 series M:10 S:4

UNC_IMC_WRITES.FULL.ANY 14,650,441,461 50,459 50,458 50,481 50,458 15,737 15,741 15,741 15,740
UNC_IMC_NORMAL_READS.ANY 14,650,441,476 196,626 196,618 196,679 196,515 36,071 36,071 36,072 36,072
----------

5.00s real 0.468s user 39.531s system 38.796s idle

Bandwidth from reads and writes is measured separately, and each processor socket is measured separately. In the above output file, the first line of values measures writes to memory and the second line of values measures reads to memory. Each line of output will show a series of event values separated by spaces. The first value after each event is a timestamp (14,650,441,461 & 14,650,441,476 in this example). The following values will be the counts of 64-byte transfers on the memory bus for each core.

It is important to realize that for current Core i7 or Xeon 5500 series processors, there are 4 cores on each socket, all sharing the same uncore. So, you will see 4 values for each socket, but really these are all measuring the same uncore bandwidth. For example, for UNC_IMC_WRITES.FULL.ANY in the example above, the first 4 values after the timestamp are all close to 50,460. They are really all measuring the same bandwidth from socket 0 to its memory, and so should be averaged, not summed! The output above was measured on a dual-socket Xeon 5500 series platform with Intel® Hyper-Threading Technology disabled. There are 8 values for each event – 4 for one socket, and 4 for the other. If Hyper-Threading Technology had been enabled, there would be 8 values per socket, and those 8 should be averaged to get one bandwidth number for each socket.

The number of values you see will correspond to the number of hardware threads on your system. The order in which the values appear may be different on Windows* and on Linux*. For Windows*, usually all the values for one processor socket will appear together. For example, on a dual-socket Windows* platform with Intel® Hyper-Threading Technology enabled, the values may be in the order , , , , , , , , , etc, giving you 8 values total for each physical socket. On Linux* the way in which the threads and cores are enumerated varies according to the distribution. You can refer to the /proc/cpuinfo file for your platform to see the way the physical sockets are mapped – for each processor in /proc/cpuinfo, look at the “physical id”. The physical id indicates the socket number. This can help you identify how the values in bandwidth.txt correspond to physical sockets (the values in bandwidth.txt will be in the same order as the processors in the /proc/cpuinfo file). In all cases – just remember that for a particular bandwidth event, you should be seeing roughly the same quantities from cores and hardware threads on the same socket. If you have a dual-socket platform with Hyper-Threading Technology enabled, then half of the values for each bandwidth event will be for each socket. Approximately half should be around the same quantity, and the other half should be a different quantity. On a single-socket platform simply average all the values.

Finally, near the bottom of each result file you will see the time spent sampling – 5 seconds in the example above.

To compute total system bandwidth, use this formula:

Bandwidth (GB/s) = ((average of UNC_IMC_WRITES.FULL.ANY for each socket + average of UNC_IMC_NORMAL_READS.ANY for each socket) * 64 * 1.0e-9) / seconds measured

For the example above, bandwidth is ((50,464 (writes on socket 1) + 15,740 (writes on socket 2) + 196,610 (reads on socket 1) + 36,072 (reads on socket 2) * 64 * 1.0e-9) / 5 = .004 GB/s. This bandwidth was measured on an idle system.

Final Notes

This method can be used to measure total system bandwidth on Core i7 and Xeon 5500 series processor-based platforms. It will not work with any other processors. We also do not recommend using sep for any other sampling – VTune analyzer and PTU have much more friendly user interfaces for collecting and interpreting data. At this time, these events (needed for bandwidth measurement) are the only uncore events we are making available.

Can’t create a new advanced build activity in Microsoft* Visual Studio* 2008

Tue, 04 Aug 2009 09:00:00 -0700

Environment:

Microsoft* Visual Studio* 2008

Intel (R) VTune(TM) Performance Analyzer v9.1 Update 2 with Intel(R) Thread Profiler 3.1 Update 12

Intel(R) Thread Checker 3.1 Update 12

Intel(R) Parallel Studio, Initial Release

Problem:

The user installed VTune(TM) Performance Analyzer with Intel(R) Thread Profiler, Intel(R) Thread Checker first, then installed Intel(R) Parallel Studio. The icons of VTune(TM) Analyzer and other tools will be hidden in Toolbar's Options due to limited length of Toolbar. However if the user opened a solution and built project(s), then click on "Tuning Browser" tab, the user can't create a new advanced build activity since the icon was grayed.

Root-cause:

This is an unexpected error of the integration of Microsoft* Visual Studio* after installing Intel(R) Parallel Studio. For example, when the user has no solution file opened, all icons of tools should be inactive; When the user opens a new solution file, all icons of Intel(R) Parallel Studio are active, but some icons of old tools are inactive.

Resolution:

Click on "Tuning Browser" tab first
Click on "Solution Explorer" tab
Click on "Tuning Browse" tab again.
Click on the project in "Tuning Browser" window

The icon of "Advanced Activity Build" should be active.

Intel® VTune™ Performance Analyzer for Windows* - Error: An error occurred during drill-down. Cause of error cannot be determined.

Mon, 01 Dec 2008 00:00:00 -0800

Symptom(s):

Selecting Don't calibrate Sample After Value (SAV) in the Configure Sampling dialog and specifying a very low SAV, so that a high number of samples per second is collected (we usually recommend 1000 samples per second), may cause the following error message to appear after sampling completes:

Error: An error occurred during drill-down. Cause of error cannot be determined.

Also, the message " The Sampling Collector failed to collect data because SAV is too low " is displayed in the Output window just before sampling data collection begins.

Cause:

Generating a high frequency of samples causes many interrupts per second to be generated. This in turn can slow the system down beyond use and can even cause the system to hang.

Solution:
Use Calibrate Sample After Value for all the selected events option to generate appropriate SAVs before running sampling data collection.

Note: This will require an additional run of the activity for each set of events to collect. The number of anticipated runs is displayed in the Configure Sampling dialog on the Events tab.

Intel® VTune™ Performance Analyzer for Windows* - Event ratios do not appear in the summary pane

Fri, 19 Sep 2008 00:00:00 -0700

Solution:
This is probably due to a missing event. Only event ratios derived from events that have been sampled and collected as part of an Activity result are available. To fix this problem:
1. Make sure to collect all the related events
2. Drag the events into the source view

Relevant Information:
event ratio

Intel® VTune™ Performance Analyzer for Windows* - Event Ratios Do Not Appear in the Source Pane

Fri, 19 Sep 2008 00:00:00 -0700

Solution:
Event Ratios are displayed in the summary pane so that the user can select a range of source lines (or assembly language statements) and see the calculated ratios for that range of code. If the user wants to see the value for an individual source line or assembly statement, that line can be highlighted and ratios will appear in the summary pane just below. Note: all the events used to calculate the ratio must have occurred on that line in order to calculate the ratio.

Relevant Information:
event ratio

Intel® VTune™ Performance Analyzer for Windows* - Unable to View the Source View when Drilling Down from the Hotspot

Fri, 19 Sep 2008 00:00:00 -0700

Solution:
Sampling requires line number and symbol information (debug information) when drilling down from a hotspot view to the source view of a module. Ensure before running an Activity configured with the sampling collector, that debug information is compiled with the binary file. If you do not supply this information, the Intel® VTune™ analyzer disassembles the binary file and displays this information in the source view. It cannot display the source code itself.

Relevant Information:
EBS, Sourceview

Intel® VTune™ Performance Analyzer for Windows* - No Connection to Remote Linux* Machine

Fri, 19 Sep 2008 00:00:00 -0700

Symptom(s):

ERROR: "Could not connect to remote machine. Connection refused. Error no. (10061)(WSAECONNREFUSED)"
This is followed by another dialog box titled "Serious Error"

Cause:
Forgot to start VTServer on the Linux side.

Solution:
Start VTServer on the Linux* Side
Other possible causes include:

TCP/IP is not running on the Linux server. Confirm that the Linux server is in multi-user mode, since TCP/IP processes don't run in single user mode (sometimes called maintenance mode).

# ps -ef | grep init

The standard output of the above command will show an init statement, followed by a bracketed number. If that number is a [1], that is, anything other than a [2] through a [5], you must bring the system into multi-user mode so that TCP/IP will be active.
TCP/IP names for the two systems are not properly set up.Try pinging between the two computers, both ways. Ping each direction first by IP address, then again by hostname. There should be no lost data packets.
TCP/IP is not properly configured on the Linux system. Confirm the TCP/IP configuration by using the following two commands:
# netstat -in
# ifconfig
Where is the non "lo" entry from the netstat command, for example:
# ifconfig eth0
Confirm IP address, netmask, and all relevant networking information. Make changes if needed, compared to the TCP/IP setup of the Windows* computer.

Relevant Information: Linux