SP1 is well worth downloading and installing - here are some of the reasons:

1. Parallel Inspector and Parallel Amplifier can be driven (for automating test suites) from the command line now.
2. Bug fixes - of course - not many issues needed fixing, but you may appreciate the ones bugs that were found and fixed!
3. Window 7 support (Parallel Studio came before Windows 7, now that it is released - we had a few things to update)
4. TBB 2.2 and other improvements to align with the upcoming Microsoft Visual Studio 2010 I'm sure there are more - these are the highlights as I see them.

Download SP1 - you'll be glad you did!

See the release notes for more details - skip the main document if you want to read about what is new and useful - read the three individual documents.

With this simple change, I notice something new in the compilation logs:

If I double-click on one of these lines in the Output panel, the system navigates to the corresponding source code lines:

Note the little marker in the left margin that indicates the lines referred by that vectorization report line. This is the ballistic step in the serial code implementation, suggesting that the little loop for setting the vector components of the bodies has been converted to a linear sequence of vector instructions. Some other time I’ll dig down into the assembly code to demonstrate that this code really has been vectorized (i.e., realized by emitting SIMD code to execute it), but for now let’s move forward and try to make this multi-thread parallel in addition to vector-parallel.

How do I “parallelize” that lower set of loops in the previous code sample? One simple way would be to add an OpenMP construct:

    #pragma omp parallel for
    for (i = 0; i < n; ++i) {
        for (int axis = 0; axis < 3; ++axis) {
            body[i].vel[axis] += body[i].acc[axis] * TIMESTEP;
            body[i].pos[axis] += body[i].vel[axis] * TIMESTEP;
            body[i].acc[axis] = 0.;
        }
    }

OpenMP has been around for a number of years and operates as a language extension for C, C++ and Fortran. Compilers enabled to recognize the constructs (such as the Intel® C++ and Fortran Compilers) can use them as hints to direct the compiler generation of parallel code. Non-complying compilers see these constructs as an unrecognized pragma (or a funny comment in Fortran) and ignore them. In this case the OpenMP line applies to the line that follows, directing the compiler to create code that divides the outer loop into some collection of chunks, each of which can be dispatched to a separate HW thread. Each thread processes the chunks assigned to it. As each thread finishes its work, it waits for the others in its team to complete their work. All these HW threads will land in a rendezvous or join point until all have arrived, because there’s an implied wait at the end of the parallel for-loop so that code that follows will not be executed until the preceding code has been completed, just to avoid any potential side effects. In this particular case, we’re also at the end of the parallel section so only one HW thread would proceed beyond the end of the for-loop, the rest returning to a thread pool to await more work.

With the advent of lambda constructs, described in the C++0x standard and implemented in the Intel C++ Compiler version 11, we can write nearly as compact a version of this parallel construct using Intel® Threading Building Blocks (before lambdas we’d need to use a full C++ function-object, which really breaks up the flow of the source code). As a lambda construct using TBB, the OpenMP code above would transform into something like this:

    parallel_for( blocked_range<int> (0,n),
      [] (const blocked_range<int> &r) {
        for (int i = r.begin(); i != r.end(); ++i)
            for (int axis = 0; axis <3; ++axis) {
                body[i].vel[axis] += body[i].acc[axis] * TIMESTEP;
                body[i].pos[axis] += body[i].vel[axis] * TIMESTEP;
                body[i].acc[axis] = 0.;
            }
      });

Not quite a compact as the OpenMP version so there must be some other reason to value this. Otherwise, why embrace the complexity? The key is flexibility. TBB offers a rich set of tools that can be used within the context of such a parallel function. The c++0x lambda-function expands that richness with a compactness of expression and flexibility that lets me use TBB with almost the same convenience of OpenMP. For example, that pair of square brackets leading off the lambda provides flexible control of what variables defined in the scope of the call will be available and in what form within the function (more on this later). The TBB parallel_for will divide the work of this inline body using as a helper class the TBB blocked_range, making work for as many HW threads as there are available.

Next time: parallel code: first runs

Working on CAD Exchanger I am designing one of its plugin to convert 3D CAD data between ACIS and Open CASCADE (two modeling kernels) to be parallel. Depending on a model size, the converter has to deal with multiple small objects allocated on a heap (e.g. 20,000+ objects each taking 48bytes + additional object data such as lists, strings, etc).

The translation works just fine and concurrency analysis with Intel Parallel Amplifier indicates high concurrency levels. So far, so good. However I had noticed that when translating the same ACIS file over and over again in the same test harness session translation took longer and longer. Why could it be ?

So I launched the Amplifier to collect hotspots and here is what I saw:

These two top hotspots relate to the memory manager layer (Standard_MMgrRaw class) which simply forwards calls to malloc/free and new/delete. Trying to root-cause the problem I had to switch to the mode to see direct OS functions (toggling off the button on the Amplifier toolbar) and here is a new screenshot:

It shows that hotspots are two system functions – RtlpFindAndCommitPages() and ZwWaitForSingleObject() – which are called from memory allocation / deallocation routines. It also shows that the nearest hotspot related to my code (BSplCLib::Bohm()) is just 1/4 of the time consumed by ZwWaitForSingleObject() (0.47s vs 1.81s).

After experimenting with several runs, analyzing how the hotspot profile changes with growing number of runs, I concluded that the first hotspot is explained by the fact that the ACIS converter creates multiple tiny objects with different size with short life span (they are destroyed after every conversion). This seems to cause strong memory fragmentation which forces the system to constantly look for new memory chunks.

The second hotspot (ZwWaitForSingleObject()) which goes through critical section is caused by the default mechanism of memory management on Windows which uses a lock.

The execution of locks&waits analysis also proves that memory management lock is the greatest one adversely affecting concurrency.

All this is caused by the direct use of calloc/malloc/free, and new/delete called dozens of thousands times. It’s worth mentioning that such hotspots did not exist when I used serial implementation and popped up only when I started using parallel one. The former used a memory manager (in a 3rd party lib) that allocated memory blocks and did not return them to the system reusing them when the application requested new blocks. I couldn’t reuse this memory manager because it was not thread-safe and therefore had to switch to another manager that simply forwarded to malloc/free.

So I almost was forced to write my own memory manager that would implement a previous behavior and would be thread-safe and … fast ! Challenges are good but not when you need to re-write low-level components what can take a lot of time and require diligent thorough testing delaying progress in your project which already receives very limited attention.

So, I approached my colleagues from the Threading Build Blocks team to check if there is anything TBB could help with. What was my surprise when they suggested me trying a new release 2.2. Version 2.2 offers a mechanism to seamlessly replace the system memory manager with the tbb allocator. ‘Seamlessly’ really means it – everything I had to is to add a single line of code into a C++ file:

#include "tbb/tbbmalloc_proxy.h"

The outcome was immediate. Not only did the hotspot profile change completely removing the OS hotspots (see the comparison mode screenshot below) but the overall speed up (on entire test case) was about 25%! One line of code, no need of re-writing anything on my own saved hours of coding, with such a return! Just incredible, the least I could say. Recently released 2.2 Update 1 includes further improvements which my app now benefits from (more reliable processing of realloc(), bug fixes for debug mode, etc).

The colleagues later explained me that the TBB allocator runs concurrently (seemingly without any locks inside) and with a similar fashion of reusing previously allocated blocks. Thus, it was the entire application (not only its parallel part) which benefited from this substitution.

So, if you are migrating from serial to parallel implementation you may encounter something unexpected – memory bottlenecks. If you got accustomed to use some nice single-threaded memory manager you can be forced to consider migration to something alternative. If this is the case you may want to give a try to tbb allocator and see if it helps in your case.

Then I ensured the vector component accumulations take advantage of the simplification:

To avoid changes to the body data structure that might affect the experiment, I redefined the acc field to mean accumulator instead of acceleration (cheap trick for a short hack ;-).

With addForce in hand, I needed to make some adjustments to the ballistic step to turn the forces back into accelerations:

Oops, adding three divides per body (one for each axis), which gives this result:

In my experiments, the add forces version came in slightly slower than my best add acceleration version. It’s easier to see in the numbers:    As this table shows, the times for the run accumulating force takes longer, as you would expect for a solution that requires more multiples (to include the extra mass term in the force equation and then to remove it to get to acceleration). But wait a minute! There’s something else going on here. What’s with those serial addAcc numbers? I remember lower numbers when I took my first serial run. Maybe there’s more variability in the results than I recall? That’s easy to check. I switched back to the bodies007 code and took several more runs.

That looks all pretty consistent across the range of n-values. Yet when I tried the same thing with bodies008:

Almost a second slower for 2K bodies, even though the supposed “code under test” didn’t change. Note: I’m not even running the addForce code!—just the conditional test in the RAMP test mode (see below). There were not many changes in going from bodies007.cpp to bodies008.cpp so it was pretty easy to isolate the code change that caused most of the slowdown. I was able to get these numbers...

…by the simple expedient of commenting out the following code:

This is one of several clauses in a for-loop that selects the values of n for the ramp; commenting out all the variant methods so that the only one remaining is the serial addAcc does even better, though the returns are diminishing:

So, given the ramp loop is somehow having an effect on the numbers, let me reverse the scenario and comment out all but the addForce variant:

Each of these last two sets are the averages of five runs each, and by my measure, adding accelerations still wins, though the answer is much more murky than I would hope. What are the gremlins that are plaguing these numbers? I have some hunches that involve optimization and inlining strategies but I can’t yet point my finger at specific problems. There are some tantalizing observations to be made, though.

For example, I could try doing a hot spot analysis to see if that would provide clues about the unexpected overhead. However, that means relaxing function inline optimization (the /Ob1 trick). But what does that do to performance?

Wow. Looks like function inlining is a big performance benefit for the serial acceleration accumulating code, in this case (one sample) having lost 14 seconds computing the interactions of 2K bodies.   Curiously, the serial code accumulating forces appears to take a much smaller hit from the loss of aggressive function inlining. Or, in glass-half-full parlance, it appears to take much less advantage of compiler optimizations.The same appears to be true if you continue relaxing optimizations, specifically looking at the performance of the Debug configuration with this same test (all these tests are using the optimizations available in Intel® Parallel Composer, so your mileage may vary depending on what compiler you use):

There’s a lot more to be discovered in this rich mine of anomalies, and perhaps when I have some more time, I will delve into it more deeply. For now though, I’ll continue to use the addAcc variant in the experiments going forward. After all, after over half a dozen posts in this series, I haven’t even gone parallel yet!

Expanding the view to show the function in detail is often called drilling down to source. In Intel® Parallel Amplifier I can do this by just double-clicking on the function, addAcc.

Parallel Amplifier lands on the hottest line in the function and provides easy navigation buttons (just below the Bottom-up button) to explore the other hot spots in order of time taken (max, step-up, step-down, min, respectively). Since I landed on the hottest hot spot, the "navigate-to-a-hotter-spot" buttons are grayed out.

Looks like addAcc has a problem with divides. Division is one of the more expensive arithmetic operations and while I can’t eliminate all of them, I certainly can reduce the number of them.

Computing the inverse distance once and then multiplying that seems to have had an effect: before the change, the times attributed to the nearby lines amounted to over 1.2 seconds while after, the total is down to 0.78 seconds. I accumulate the values from the adjacent lines because the reported event counts are at best approximate—the tool needs to deal with both the optimized code that may have scattered around the instructions that implement any particular line and phenomena that affect the actual process of determining the location of the instruction pointer, such as event skid (to be addressed in some other post). In fact, probably a significant portion of the 0.78 and 1.2 seconds is actually coming from the square root function that immediately precedes these lines. So I’ll run another ramp of n-bodies and see if my numbers are any better.

Yes, they are. And now that I have ivdist, it’s worth considering whether I can use it more efficiently to replace the divide by distsq into something like this:

    double Gdivd = GFORCE * ivdist * ivdist;

Sure enough, that change, though not as beneficial as the last one, still has an observable benefit:

Another hot spot run shows even less time being spent at the previously identified hot spots:

I’ll use this last version as my serial baseline, the benchmark against which I’ll compare to measure my progress in parallelization. It might change someday as I continue to evaluate alternatives like the persistent question about forces. This in fact is part of our recommended practice for migrating serial code into a parallel environment: I start by optimizing the serial version as much as I can so that the benefits I gain through parallel implementation are not just because multiple HW threads are just filling in the gaps left behind by reusing an inefficient, serial version.

Next time: parallel code, a first attempt

It installs right in Visual Studio as shown above. In order to collect hot spots on the serial algorithm, I switch the debug command to single 256 serial

I’ve also turned on symbols in my Release configuration (C/C++ >> General >> Debug Information Format set to and Linker >> Debugging >> Generate Debug Info set to on my latest build), then just click on the Profile button, and viola!

Huhhhhhh?! I see two seconds plus a quarter spent in main, but where are my functions? Do I get the same result if I try the Debug configuration?

Oh, there are my functions, runSerialBodies and addAcc, but the run takes over 5 seconds. I don’t want to spend time making Debug code run faster, so I want to tune the optimized Release code. However, something about that Release configuration is causing the functions to disappear. Experimenting a little with the configuration settings reveals that the Intel compiler is automatically inlining the functions into main. Unfortunately, apparently there’s no way to represent that inlining in the debug information so the functions just disappear. By relaxing the optimization a little, I can restore the function hierarchy for analysis at the cost of some extra function call instructions:

Now my hot spot analysis on the Release configuration looks much better:

Most of the time is being spent in the addAcc function, which is being called by runSerialBodies as can be seen in the function call hierarchy graph. Looks like addAcc will be one of my candidates for parallelization.

Next time: serial body drill-down

Download the video.

Download and MP3 of the show.

Today on the show we'll be speaking with Roderick Kennedy, President and CEO of Simul. Simul is a software company specializing in innovative, lightweight solutions inspired by physics based in Manchester, England. Roderick is a specialist in game physics and simulation, has worked in the games industry since 1990, contributing to such titles as DID’s Eurofighter 2000, and Evolution Studios’ World Rally Championship series.

First the News:

Intel Threading Challenge PHASE 2

• Problem 1 - "Strassen's Algorithm" winner was iArchitect
• Problem 2 - "Knights Tour" is CLOSED for submissions!
• Problem 3 - "Graph Coloring" went live Sept 21 and is due October 9th

The University of Illinois is presenting a lecture on-line this Friday
XcalableMP: A Performance-Aware Scalable Parallel Programming Language for Distributed Memory System - Beyond PGAS Models Mitsuhisa Sato, Director for the Center for Computational Sciences, University of Tsukuba Friday, October 2, 2009 at 2:00 PM (Central Time) at 2405 Siebel Center for Computer Science.

Live video streaming*: http://media.cs.uiuc.edu/live/upcrc0910/upcrc.asx

SC09 – Super Computer Conference is coming up November 14-20 in Portland, OR. The Intel Software network team will be participating in panels and on the show floor for the whole event.

Listener Questions Show is the first Tuesday of each month. October 6th is the next one. If you have a question or idea about the show send it in to ParallelProgrammingTalk@Intel.com

Intel Developer Forum was September 22-24. Our friends at Softtalkblog did a great job highlighting many of the great parallel programming sessions. Here are a  few excerpts:

In exciting news for developers, Intel CEO Paul Otellini has just announced the Intel Atom Developer Programme here at IDF2009 in San Francisco. The programme is aimed at developers who want to create new apps or port existing ones to Atom, enabling them to enter the vibrant market of internet-enabled devices. Intel will support developers with software development kits, technical support and community resources. Developers will also be able to trade code components (buy and sell), so they can accelerate their development time, and receive income from landmark innovations early, before they are part of an end product.

There were many technical sessions on Parallel Programming Technology and tools.

In the "Go Parallel" session James Reinders explain the language and library support available for developers today and compare it with what the future holds. Intel is now working on a language extension, based on Cilk++.

Intel Concurrent Collections now offers Linux support as well as a preliminary implementation for Haskell.

Victoria Gromova, one of Intel’s senior software engineers, explaining how developers can achieve forward scalability with Threading Building Blocks (TBB). Having been around for some time now, TBB is an Intel Open Source project, dedicated to facilitating parallel programming by providing a library of template classes and functions for C++ developers. In doing so, TBB helps developers achieve the two ‘holy grails’ of parallelism: correctness and performance.

Steve Teixera, product unit manager of parallel development tools at Microsoft, presented a session on the future of parallel programming with Intel Parallel Studio and Microsoft Visual Studio.

As many of you know there are many challenges to building parallel applications. It needs to be much easier, and people need to know how to tune their applications to make them efficient.

Steve outlined some of the ways that Intel and Microsoft have been cooperating to make multicore programming easier. Firstly, the concurrency runtime in Windows 7 helps to avoid resource conflicts between Intel Threading Building Blocks, Open-MP and Microsoft Parallel Pattern Library. Secondly, Intel Threading Building Blocks and Microsoft Parallel Pattern Library now share a common data structure for vector models. Thirdly, the concurrency runtime scheduler in Windows 7 enables load balancing, using work stealing so that a processor will take work from another processor’s queue if it has immediate capacity.

In another session, Mark Davis and Ravi Vemuri both with Intel, demonstrated how the Intel Parallel Advisor Lite can be used to model parallelism in serial code, to ease the transition towards parallel code. Intel Parallel Advisor Lite is in preview, and we invites you to download it, use it and submit your feedback. You can get a free copy at whatif.intel.com. You will need Intel Parallel Studio, but can use an evaluation edition

There were many other technical presentation and Q&A sessions. Learn more about IDF, watch the videos and download the presentations at http://www.intel.com/idf.

Today's Show:

Today on the show we'll be speaking with Roderick Kennedy, President and CEO of Simul. Simul is a software company specializing in innovative, lightweight solutions inspired by physics based in Manchester, England.

Roderick talked about his company, the market for his technology, and the process they used to multi-thread their application. We discussed his evaluation of OpenMP vs. Threading Building Blocks and his eventual decision to use Threading Building Blocks. Listen or watch the full show to understand their challenges and the impact on Simul Software's weather application.  Download a case study and visit the Simul Software web site to learn more about the company and multi-threading positive impact on application performance.

Coming Up Next on Parallel Programming Talk:

The first Tuesday of the month is our Listener Questions show. October 6th is the next one. If you have a question or idea about the show send it in to ParallelProgrammingTalk@Intel.com

Oops, that’s right. bodies007.cpp relies on language extensions available in the Intel® Compiler version 11, some early arrivals from the C++0x standard. Fortunately, it’s pretty easy to switch compilers.

With the Intel C++ Compiler installed in Visual Studio from either of the regular distribution packages, the Compiler Professional Edition or Intel Parallel Composer, switching compilers is just a click away.

OK, two clicks. I did try to build already, so I’ll let the system clean up the project files.

Viola!!! The project now uses the Intel C++ compiler.

One more configuration setting to enable C++0x support:

And now it compiles!

Dropping the serial code into the test program I’ve prepared gives access to a simple command processor that allows me to select that kernel for one of several tests. A simple ramp, testing the algorithm with varying values of n, can be launched by putting this in the command line: select serial

Looks like the time it takes to complete the simulation (I’m running 1000 time steps for each body-count) is going up more than four times for every doubling of the number of bodies; in fact, if I plot it on a logarithmic scale, I can see the exponential growth:

Clearly this is an algorithm that has plenty of work to divide, if I can just figure out a way to do it.

Next time: serial body hot spots

i and j are indices selecting elements of the body array. First task is to compute the distance between them.

Pythagorean Theorem in three dimensions gets me the square of the hypotenuse, but before doing the square root, I’ll avoid the singularity:

That is, if the point masses get too close together, act like they’re not. But wait! Why do I even need the square root, if I’m working with gravitation, an inverse-squared law? Well, because acceleration is a vector so I need the next step.

Array ud represents the unit vector (length 1 direction vector) pointing from body j to body i. I need just one more thing, the magnitude of those accelerations.

All that’s left is to compute the acceleration vector components and apply them to the bodies.

Next time: serial bodies test run

Today, at Intel's Developer Forum, we have taught many classes on our tools, and have a few left to go.

If you could not join us in San Francisco, the presentations are available online for downloading at intel.com/go/idfsessions.

My talks, including one today with Steve Teixeira of Microsoft, can be found searching for LAST NAME of "Reinders." Today's class was a lot of fun - it was fun to share the good work we are doing together, and Steve introduced me to the concept of "clinging to curly braces" in the course of an engaging Q&A.

Presentations on Intel Software Development Products that are available online (after the presentations are done):

Even better is if I can squeeze those inefficiencies out of the algorithm itself. Previously, before defining the interaction loops, I set up a data structure to represent each body, choosing mass, location, and velocity to represent the state, plus a place to accumulate accelerations. However, Jim Dempsey suggested it might be more efficient to accumulate forces rather than accelerations. Forces are the canonical result of the gravitational equation but divides are one of the most expensive floating point operations. Better to accumulate the influence of other bodies as forces first, then do one divide to compute the acceleration, right?

But where did that F come from?  We are actually dividing out a mass that was used to compute the original force. What if we never multiplied it in the first place?

The G-over-R-squared is a common factor that can be computed once. Two multiplies, which would be required anyway, give the accelerations directly, without any extra divides!   So we'll stick with the plan to add accelerations.

Next time: realizing addAcc(i,j)

// typedef tbb::concurrent_vector< std::pair<int,int> > cv_t;
cv_t::iterator it = cv.push_back( item ); // add new item
map[ item.first ] = &( *it ); // store its address in a table

For grow_to_at_least() which previously returned nothing, there is additional benefit of delimiting a range that was allocated and initialized by this particular call and thread. E.g.:

cv_t cv( 100 ); // initial 100 items
assert( &( *cv.grow_to_at_least(50) ) == &cv[50] ); // doesn't allocate items
assert( &( *cv.grow_to_at_least(250) ) == &cv[100] ); // allocates additional 150 items

The vector contains 100 items. grow_to_at_least(50) will return an iterator pointing to an item with index = 50. But sequential grow_to_at_least(250) will return iterator which points to item #100. It means that additional 150 (= 250 - 100) items were added by this call.
It makes more sense in concurrent environment, as these items were correctly initialized but not just allocated. The same guarantee is already provided for grow_by() and all its items.
If you wonder how items may be allocated but not initialized, please read my blog which clarifies semantics of size() and grow_to_at_least() which were also fixed in 2.2 to allow safe parallel inspections. BTW, tbb::zero_allocator was added to support use cases from this blog.

In order to align with std::vector from C++0x, method compact() was renamed as shrink_to_fit(). However, it doesn't suggest any change to semantics and practical means of the method. It still will defragment the first segments as described in my another blog, and it will not change capacity exactly to the value of size().

To close the topic of interface changes, I'd remind you that you might compile old programs with new TBB as is just by setting TBB_DEPRECATED=1 as described in transition blog.

Another part of the changes is hidden behind implementation and the fact that exceptions are rare for concurrent_vector. So, a few chronic critical bugs of exception safety were fixed to comply with declared guarantees. However, these fixes miss the 2.2 release and available only in consequent stable and update releases.
So, please consider upgrading to the latest TBB after the 2.2 release in order to avoid a deadlock inside concurrent_vector which may occur for e.g. "out of memory" exception in all the previous versions.