Where do you go to learn about the features of Intel Software Development Products?    Soon the answer to this question will be Tool Talk.   Let me tell you about a new program we are starting.   We are creating a new video series called Tool Talk.  I will host these episodes.   Each segment will be about 10 minutes long.  I will introduce a feature of one of the Intel Software Development Products and explain how to use it and what it does.   Feel free to add your comments or come over here to my blog and ask a question.    A week or two after I talk about and demonstrate a feature of Intel Software Development Products I will host one of the Intel lead engineers or architects to talk about the technology or what goes on underneath the hood when you exercise that option.   

The first session is already live!   View it here:  Tool Talk Episode 1 

In this first session I talk about threading for multi-core platforms with Intel® Cilk™Plus.   Using scalable threading abstractions such as Intel® Cilk™Plus or Intel® Threading Building Blocks is a much easier way to express parallelism and write maintainable code than it is to explictly create and manage threads on your own.    I can not cover everything so if you have a specific question post it here for me or come to the Software Product Forums and start a discussion.  Just select the Intel Software Products header and the appropriate section to begin a new thread. 

Let me know what topics/features you would like me to cover in a future episode.

-David

-David

Today Douglas Armstrong, Intel® VTune™ Amplifier XE architect, joins me to share tips on using VTune Amplifier XE for tuning TBB software.   VTune Amplifier XE has built-in support for helping find and tune the granularity of domain decomposition in TBB.   Douglas feels this is an under-appreciated feature and has captured some screen shots to share with us.   Douglas created a sample TBB application and analyzed it with the concurrency option of VTune Amplifier XE.   Below we show a screenshot of the summary view.  It appears as though we were almost fully parallel the entire time.

Summary concurrency view

You may think this means we have almost perfect threading as there are not blocked threads or contention on synchronization. But as we scroll through the full summary display, you'll see the the elapsed time of over 8 1/2 seconds. There is also a new metric here called "Overhead Time" that indicates over 1 1/2 seconds, over 10% of the total CPU time of the app was spent on synchronization overhead. You can even see this CPU time being spent in the TBB internals in the list of the top hotspots of the app. Amplifier XE is even putting up a warning note here saying that we may have a problem with synchronization overhead.

Overhead Summary

Let’s take a look at the results in the “Bottom-up” tab to see what’s going on. If we just look at where the CPU Time is being spent you’ll see that it is all labeled in green, meaning that we were fully utilizing the system’s processors when it was running. About 14 seconds looks productive in do_work. But, notice that some of CPU Time is showing up in TBB functions, this time is also marked as Overhead Time.

Overhead Time

If we arrange the data by module, instead of the default  arrangement by function, we can see that there is three fourths of a second of overhead time in the TBB DLL, but there is also almost a second of overhead time in the user module. This is because the TBB header files include many templates which cause inline functions to spend time working on TBB overhead within the user modules. If we arranged the data by source file this is confirmed.

Module Base view

Arranged by Source File

So now that we know we have lots of overhead from using TBB, how do we fix it. Let’s start by figuring out where that overhead is coming from. This test application uses TBB for a couple of different algorithms so we need to find out which one is causing the problem. By looking at the function with the most overhead time we see it is labeled “[TBB parallel_for on class inner_body]”. This is Amplifier XE’s way of telling us that the time was spent processing for a TBB parallel_for template based on the class “inner_body”. We could go check out how we are using that in our source code. By selecting that row, the call stack pane on the very right shows the call stacks associated with this selected overhead. Make sure that the stack type selector combo box is set to “CPU Time” instead of “Wait Time” when looking for overhead. The third row down here refers to line of source code where we called parallel_for() in this inner_body class. We can double click on that and see the actual source code.

Concurrency Expanded

Source View (Drill Down From Expanded Concurrency)

By looking further we find out that a grainsize of 1 was manually specified for this blocked_range with over 8,000 iterations. The appropriate grainsize is based on finding the right balance of doing enough cycles in the body so the ratio of work to overhead is low while still allowing TBB the flexibility it needs to schedule work on different threads to keep them busy. This grain size is too small. A bad setting in the other direction, a grain size too large, could cause imbalance problems with TBB being unable to break up the work and schedule a pieces to another thread. This could show up in Amplifier XE as poor utilization of CPUs.

For now, let’s fix this grainsize to a larger number and see the result. From the summary, it now looks like we have removed this significant TBB overhead and the elapsed time has dropped from 8.7 seconds to 6.9 seconds. So we have used VTune™ Amplifier XE to help us identify poor grainsize selection and improve performance of this software. Let us know if you have other examples you would like to see illustrated with VTune™ Amplifier XE.

Concurrency Summary View after Tuning

*All data was collected on a laptop with an Intel® Core™ 2 Duo processor running Microsoft Windows* 7 using Microsoft Visual Studio 2008 SP1, and Intel® VTune™Amplifier XE 2011.

If you missed the news about Intel Parallel Studio 2011 then click here to take a look.  Intel Parallel Advisor is a brand new feature to help identify and model opportunities for parallelism.  

* trademarks

Check out the contest web page for full information about contest details and prizes.

Vasanth Tovinkere, one of the software engineers on my team who is great at parallelization and software tuning will be conducting this web seminar.  This presentation will be mixed with live demonstrations that show how easy it is to leverage the power of Microsoft Visual Studio and Intel® Parallel Studio to find performance issues due to lock contention in threaded applications. This ensures that shipped applications can take better advantage of the available processors in end user systems. Intel Parallel Studio is an add-in to Visual Studio to help create fast, reliable code that takes advantage of multicore processors.

The web seminar is Thursday 18 February 2010 at 9:00 AM PST.  Go here to sign up and register!

We have more good news for you too!   The marketing team worked with the Intel Software reseller Programmers Paradise to help sponsor the event and so the course will be free to attendees.    There will be a variety of prizes to those who attend.    Events are scheduled for Houston TX, Austin TX, Iselin NJ, New York NY, and Waltham MA in February and March.   Follow the links, register and come!    Keep track as we may add more cities in the future. 

The second web seminar I mention was also in December.   I hosted this intel web seminar and Matt Dunbar the director of performance technology at Simular presented. This web seminar was part of the Intel Real World Parallelism series.    Just follow the link here. Matt Dunbar spoke about the Simulia product and how they use Intel Parallel Inspector in their development process.  A couple of points Matt made are listed below:

• Their code base has over 1 million lines of code.  They had Intel Parallel Inspector up and running from within Visual Studio and command line interface within one hour.
• With threading error analysis in Parallel Studio, it is possible to write simpler component tests and have errors trapped

Memory analysis and thread data race detection are complex analysis operations.  Using multiple small component like tests match our recommended usage model and it was great to hear Matt recommend similar ideas.   Matt also talks about their usage in interactive mode and in batchmode.  Just select the "Real World Parallelism" tab on the web site.  Check the box " How to Use Intel^® Parallel Studio to Streamline Code Development in a Multicore Environment - presented by SIMULIA."

The Intel web seminar requires free registration.  I hope you find them interesting; if so download an evaluation copy of Intel(r) Parallel Amplifier and try it out.

Intel Parallel Amplifier provides 3 basic analysis types: hotspots (with call stack data), concurrency and locks  and waits.  The locks and waits analysis highlights which synchronization objects block threads the longest.    It is common for software to have too many or too few synchronization points.   Insufficient synchronization points lead to race conditions and indeterminant results (if you have this problem you need Intel Parallel Inspector, not Intel Parallel Amplifier.  See this MSDN web seminar for more on Parallel Inspector:  Got Memory Leaks? ).  If you have too many synchronization objects you want to know which ones if removed would improve performance the most.   Even if all the synchronization objects you have are necessary, you might want to know how much they impact performance. This may help you decide whether to spend time to refactor the software so the synchronization object can be removed.     The Locks and Waits analysis in Parallel Amplifier tracks how much time threads spend waiting on each synch object and reports the times in an ordered table.  The synch objects that cause the most waiting are listed at the top, the rest in declining order.   

Let’s look at a simple example.  One of the commonly used computational exercises to teach parallel programming is a simple program to calculate pi.    The main process thread creates NTHREADS threads and sets them off to execute a routine called calcpi, then waits for them to complete.   The basic algorithm in calcpi is shown below: 

47       int my_begin = myid * (ITER/NTHREADS) ;
48       int my_end = (myid +1) * (ITER/NTHREADS) ;
49       step = 1.0 / (double) ITER ;
50       for (int i = my_begin ; i < my_end ; ++i)
51       {
52           x = (i+0.5) * step ;
53           EnterCriticalSection(&protect);
54           pi = pi + 4.0/(1.0 + x*x) * step ;
55           LeaveCriticalSection(&protect) ;
56       }

 Each thread takes a section of the for loop to operate on.  Pi is a global variable so when each thread updates pi, a critical section protects its access from any data races.   After creating NTHREADS and assigning them to begin executing function calcpi, the main thread waits for all the threads to complete.    When I run this through the Locks and Waits analysis of Intel Parallel Amplifier I see three synch objects appear in the analysis section. This screenshot is shown below.     I can expand each of these three synch objects and drill down to the source code.   The top synch object where a thread waits the most time is the main thread waiting for all the threads it created to complete.    All this thread does is create threads and wait.    There are no intermediate steps to tune.  We certainly don’t want to exit the program before calculations are done, so let’s keep this synchronization point.    Let’s look down the list for the next opportunity.         

 The The second item in the table highlights a critical section.This second synch object that shows significant wait time is the critical section in calcpi.    I can double click and drill down to source and verify that the critical section that causes so much delay is the critical section shown in the code segment above.  In the screenshot, you can also see this was line 53 in my code (I rearranged the display so the creation line column is next to the wait time   You may rearrange column ordering as you like, by dragging each column to the desired position). So let’s look at how I can minimize the number of entrances into a critical section.  The contention is around the access to this global variable pi.   There is no reason that I must accumulate each contribution in the global variable.  I can create partial sums within each thread and combine these partial sums for the final result.   The first thing I do is create a local copy of pi for each thread.  I call this my_pi.  Then each thread calculates my_pi and when the for loop is complete, each thread adds their portion of the calculation into the global variable pi.   For the data collected here, I used a dual core system and 1,200,000 iterations with NTHREADS set to 2.   So for the first algorithm I entered a critical section 1,200,000 times.  For the algorithm below, I enter the critical section only twice.  

50         double my_pi = 0.0 ;
51         for (int i = my_begin ; i < my_end ; ++i)
52        {
53             x = (i+0.5) * step ;
54             my_pi = my_pi + 4.0/(1.0 + x*x) ;
55         }
56         my_pi *= step ;
57         EnterCriticalSection(&protect);
58         pi += my_pi ;
59         LeaveCriticalSection(&protect) ;

  As a further optimization, I pulled out the multiplication-by step out of the for-loop and did the multiplication-by step once per thread rather than once per iteration.   Now when I run this through Parallel Amplifier the screenshot is shown below.

Once again the top of the table is the thread in main waiting for all the worker threads to complete.      The code runs so much faster the scale of the display changed from seconds to milliseconds.  The amount of time spent waiting is far shorter.       The second entry in this screenshot  (Stream)   maps back to a print statement – remember printf is an implicit barrier and its behavior is tracked just like the calls to critical sections or WaitforMultipleObjects.   

So what to do next?  Get an evaluation copy of Intel Parallel Amplifier and analyze your code.  Check out which synch objects have threads waiting the most.  Then see if you can find some things to optimize.  The difference in runtime between the two algorithms I showed here is about 50x.  This was a simple example, where the worst case performed worse than the sequential algorithm, so changes in performance of this magnitude are not common.  Improved performance by minimizing time spent waiting at barriers is common, though.    So see what you can get.   Now, if I had I begun with Intel Threading  Building Blocks, I would have let the Threading Building Blocks parallel reduce algorithm handle the reduction intelligently and I would have had a very different experience, but that would be a topic for a different blog.

Intel Parallel Inspector checks for both memory issues and threads data races and deadlocks.   This new product uses a new interface and we believe is easy to configure and use.   I will be conducting a web seminar on Tuesday 8 December at 9:00 AM Pacific time.    I will cover both the memory inspection feature as well as the thread inspection aspects of Intel Parallel Amplifier.   The link to register to attend this event is listed here.   I hope you can join me at this MSDN web seminar.

In addition to presenting at the MSDN web seminar I get to host the Intel web seminars.   I hope you get to catch these in the spring and the fall.   This fall we did something new and brought in several ISVs to tell about their experiences threading for Intel multi-core platforms.    This series is called Real World Parallelism.    Last week I also get to host Bernard Laberge, Senior principal Engineer in the video editors division at Avid.    There is one more web seminar in this series on 15 December.   The old seminars are available for playback - just check the Intel software products web pages here.  Come join me and Matt Dunbar  on 15 December 15

-David

The events are different for each platform so you need to know what platform you are executing your software on.   We just recently updated our guide to event based tuning on the popular Intel(R) Core(TM) i7 processor based platforms.  The guide is updated to include suggestions for both single socket and dual socket systems.   Check out:  http://software.intel.com/en-us/articles/using-intel-vtune-performance-analyzer-to-optimize-software-for-the-intelr-coretm-i7-processor-family/

Many of you following our platforms know that on our new platforms the memory bandwidth measurements can no longer be done on the core processor.    For those who want to measure bandwidth on the Intel(r) Core(TM) i7 processor based platforms we have enabled a method using uncore events.   Check out this link:  http://software.intel.com/en-us/articles/how-do-i-measure-memory-bandwidth-on-an-intel-core-i7-or-xeon-5500-series-platform-using-intel-vtune-performance-analyzer/  You must have a VTune Performance Analyzer License and you will also need to download the Performance Tuning Utility from whatif.intel.com.

*Other brands and names are the property of their respective owners

Does it mean there is more for software developers to do now? Maybe (undoubtedly there are more ways to make mistakes)? Is it a lot more to do or are there different ways to do things? This is why I preferred to refer to another of Herb Sutter's article where he called the changes going on a concurrency revolution (see http://www.ddj.com/architect/191800187). I used to have a copy of the last couple of paragraphs from this article taped to the outside of my cubicle wall (when we moved around it went away and I did not repost it). I quote a couple of sentences from his article: " Yes, this is a wonderful time to be a software engineer. For the rest of this decade and into part of the next, we're going to do for concurrency what we did for objects and garbage collection: ..."

Rather than thinking of working for lunch now, I like to think of being on the forefront of a transition or revolution (Yes, I like parallel programming. That is what I was doing when I joined Intel in 1992 and when I had a chance to do it at Intel again I took it). There was a big shift years ago to object oriented programming. There is a shift now to concurrency. Did the shift to object oriented programming mean more work than procedural programming styles? I don't think so. But if you had a lot of legacy code the transition wasn't free. Likewise, the transition to concurrency allows for more complexity, and can be more difficult. If we don't design well and use the best tools, we can run into paralyzation instead of parallelization. If we design well and use the best techniques we can minimize the complexity.

One of the reasons Intel created the Threading Building Blocks is to provide an abstraction for threading that provides good performance and removes the developer from the tyranny of trying to explicitly create and manage all of the threads. The developer can focus on the parallel tasks and templates and allow the runtime library to manage scheduling and management of the threads. We also wanted something that was evolutionary – something that works in an environment that is familiar to developers without requiring them to learn a new language or use a special compiler. C++ templates fit that perfectly. We wanted this to help drive a change in the way developers think and develop – that is make thinking parallel a natural part of software development – so we released this as an opensource project (threadingbuildingblocks.org).

We have seen great responses from and adoption by many ISVs (e.g. Autodesk Maya http://softwarecommunity.intel.com/isn/downloads/softwareproducts/pdfs/Brief_TBB.pdf). When my team of engineers visits software developers we sometimes get the response –" that is exactly what I need (parallel pipeline, or concurrent hashmap, or...), Threading Building Blocks will be easier for me than me implementing that by myself, I will just adopt Threading Building Blocks for that". Yes we sometimes meet developers who say, I have already implemented threading with native Windows threads or Posix threads and don't want to change.

One way to look at how the environment is changing is to look at not just how the development community picks up Threading Building Blocks beyond the platforms we developed it on. One of the more recent announcements though was adoption of Threading Building Blocks by Deep Shadows. Now their port was not an academic exercise of will this work well enough to run a benchmark. They wanted to use this in their product (see http://www.developmag.com/news/30160/Deep-Shadows-talks-multiprocessor-optimisation). One of the platforms they develop for is the Xbox game console. By completing a port of Intel Threading Building Blocks to Xbox they can get the parallel performance they want and maintain a more common code base between Xbox and PC versions of a game. They generously contributed that back to the threading building blocks community. Because of their contribution any Xbox developer can download the threading building blocks source to use threading building blocks on Xbox. This is not the only port that the community has contributed. Others have contributed ports for PowerPC Linux.

This summer we launched Intel® Threading Building Blocks 2.1. This included several improvements to increase performance and to clean things up. Several new items were added in response to feedback we received in the Threading Building Blocks forum – for example parallel_do. We appreciate the response and feedback we receive in the threading building blocks forum and in our face to face visits with software developers.

Herb Sutter wrote in the article I referenced above that "The world awaits." Let me ask what are you waiting for? If you are not threading for performance now, what types of abstractions or development products are you looking for or waiting for? What else belongs in Intel Threading Building Blocks? What do you want in the developer's toolkit that is not available now?