To facilitate interoperation of compilers and such tools, we have posted an open specification open specification of a feature for communicating such events from the compiled code to the analysis tool.  The feature consists of two compiler intrinsics and an extension to the executable format.  We're already using the feature in:

• The Intel compiler implementation of Cilk Plus.
• Intel® Inspector
• Intel® Cilk™ screen and Intel® Cilk™ view.

and of course working on adding it to the Cilk Plus development branch of GCC, so that code generated by that GCC branch will be analyzable by the aforementioned Intel tools.  See the link "Intrinsics for Low Overhead Tool Annotations" on the page Intel^® Cilk™ Plus Specification for the specification. 

The feature is not limited to Cilk Plus -- it's a general mechanism for communicating events.  Better yet, it exists as a compiler intrinsic function, so that you can mark up your own code with your own events, and not just rely on ones created by the compiler.  Of course you’ll need an analysis tool that understands your events.  We’re working on a library to make it easy for users of  Pin to decode the events.

The feature avoids the overhead of the traditional approach of reporting an event with a subroutine call, which incurs run-time overhead even if the subroutine does nothing when the program is not being analyzed.  The feature creates a table in the executable of where the events are, and optionally a no-op instruction that can be overwritten by a tool like Pin if you need to instrument the event.  We've been careful to design the extension to the executable format so that only the compiler and analysis tool need to understand it.  Other parts of the tool chain such as the assembler and linker do not have to understand it, unless they attempt a level of code rewriting not normally seen in such tools.

The specification is intended to enable compilers to generate the annotations and program analysis tools to consume the annotations.   As an example, it shows how we use the annotation mechanism to mark Intel® Cilk™ Plus programs. 

So if you are interested in program analysis tools, please take a look at the specification.

More Efficient Handling of Array Sections

Suppose p and q are pointers.  Consider the following array-section assignment:

p[0:n] = q[0:n]+1;

This statement sets p[i]=q[i]+1 for i in {0...n-1}.  But what happens if these two sequences overlap in memory?   In our original specification, we followed the practice of APL and Fortran 90, and said that the right side must be evaluated first before it is assigned to the left side.  This practice makes the example well defined regardless of whether there is overlap.

However, a key principle of C++ is "abstraction with minimal penalty".  The APL and Fortran 90 approach violates this principle, because it requires the compiler to generate a temporary array for the right side result whenever the compiler cannot prove that there is no overlap.  In many practical situations, the compiler cannot prove absence of overlap, even though the programmer knows there is no overlap.   Thus with the old specification, the programmer ended up paying a penalty for using array notation.

The revised specificaiton makes the partial overlap case undefined.  The compiler is free to compile the example without generating a temporary array.  Indeed, under some circumstances, the the array sections can conceivably be compiled into more efficient code than their Fortran counterparts.  Now it is a programmer error if p and q point to partially overlapping arrays.  However, if the overlap is exact, the code is well defined.  This is important for keeping the equivalence of common update idioms like "p[0:n]+=1" and "p[0:n] = p[0:n]+1". 

The new specification makes the overlap rules for array sections consistent with the existing ones for structs/classes/unions.  If I write "*p = *q" and the pointers point to structs, the assignment is well defined only if p and q point to non-overlapping structures, or point to exactly the same structure.   That way, the compiler can avoid generating unnecessary temporary structures.  Now the rule and its benefits apply to array sections too.

The change will break existing code  that relied on the temporary arrays.  It's an inconvenience that in some cases might require revising existing code, but I believe it's justified by the greater long-term good for Array Notation.

Rank of "Pointer + Int" Resolved

In Array Notation, "x op y" is normally performed elementwise if x or y is an array section.  For example, "z[0:n] = x[0:n]+y[0:n];" sets z[i] to x[i]+y[i] for i in {0..n-1}.   One-dimensional sections are said to have rank 1.  The rule extends to multidimensional arrays.  For instance, "a[0:m][0:n] = x[0:m][0:n] + y[0:m][0:n];" does elemntwise addition and assignment across two dimensions.  The array sections are said to have rank 2.  In general, in "x op y", if both x and y have non-zero rank, they must match, and the rank of the result is the same. 

But there is a an important exception for the subscript operator.  Suppose x and i are expressions with non-zero rank.  The rank of x[i] must be the sum of the ranks of x and i.  This is because expressions like x[0:m][0:n] are equivalent to (x[0:m])[0:n].  That is, subscripting is always a one-dimensional operation in C++, and multiple dimensions are "faked" by multiple subscript operations. 

So far, this is plain stuff, and seems non-controversial.  The tricky issue is determining the rank of p+i when p is a pointer and i is an integer, both with non-zero rank.  Should it follow the normal "x op y" rule, or the array subscript rule?  Either way breaks one of two fundamental identities in C++:

1. (p+i)+j == p+(i+j)
2. p[i] == *(p+i)

Following the "x op y" rule preserves identity 1, but breaks identity 2.  To see this, suppose p, i, and j are expressions with rank 1.  Then both sides of identity 1 have rank 1 under the "x op y" rule.  But in identity 2, the sides have different rank: "p[i]" has rank 2 and "*(p+i)" has rank 1.

An alternative would be to treat "p+i" similarly to "p[i]", and say that the rank is the sum of the ranks of the operands.  That preserves identity 2.  But it breaks 1, because then "(p+i)+j" has rank 3, but "p+(i+j)" has rank 2.

So one of the identities must break.  The tie-breaker is consideration of templates and possible future extension to user-defined operators.  In those cases, we want a uniform rule for "x + y" that does not depend on the types of x and y.  So we broke identity 2, and kept rank of "p+i" is the same as for any other "p op i".  Subscripting is the only special operator with respect to rank.

Summary

The revised specification resolves some important issues in Array Notation.  It should make it much clearer to both users and implementers, and enable "abstraction with minimal penalty" for Array Notation.

I needed to write a routine similar to std::copy, except that the routine needed to destroy its source sequence; i.e., revert it to raw memory.  I was amused that I could exploit both C++ 2011 and the Intel Array Notation extension to give the compiler wider license to optimize the code.   Here is the code: 

template<typename T>
T* destructive_move( T* first, T* last, T* output ) {
    size_t n = last-first;
    []( T& in, T& out ){
        out = std::move(in);
        in.~T();
    }( first[0:n], output[0:n] );
    return output+n;
}

I used the Intel 12.0 compiler and Microsoft Visual Studio 2010 to compile it.  The code combines three new features:

• A C++11 lambda expression:  That's the code introduced by  [], and creates a functor object.  The functor has two formal parameters in and out.
• A C++11 "move assignment": It moves the contents of in to out, and gives license to operator= to mangle the value of in for sake of optimization.   This can be a significant performance advantage over a C++98 copy assignment  "out = in".  For example, if T is a std::vector<U> with N elements, a "copy assignment" of T takes O(N) time, but "move assignment" takes only O(1) time, because it can steal the underlying hunk of N elements instead of copying them.  Furthermore, often "move assignment" can be guaranteed to not throw an exception, because it steals resources instead of allocating more resources that might run out.
• Intel®  Cilk Plus Array Notation: It maps the functor over first[0:n] and output[0:n].  Each of these represents a sequence of n elements, starting at addresses first and output respectively.  Array Notation is an Intel extension that avoids the need for an explicit loop, and gives the compiler license to vectorize the code.   The only drawback is that if an exception is thrown, it's not clear how many elements of in[o:n] were actually destroyed.  Though for the cases of interest to me, neither the "move assignment" nor explicit destructor ~T() can throw an exception. 

I could have written the routine as:

template<typename T>
T* destructive_move( T* first, T* last, T* output ) {
    size_t n = last-first;
    output[0:n] = std::move(first[0:n]);
    first[0:n].~T();
    return output+n;
}

which is more concise, but has the drawback of making two sweeps over memory.

More about Cilk Plus can be found here.

Hyperobjects are Cilk's way of doing parallel reductions.  The best reference on them is the award winning paper "Reducers and Other Cilk++ Hyperobjects".   I'll summarize their proper use as background for their abuse. 

Intel® Cilk™ Plus allows control flow to fork into multiple strands of execution.  A hyperobject is a special kind of object for which there are multiple views.  Any two concurrently executing strands get separate views that they can safely update without locking.  Here is a trivial example:

#include <cilk/cilk.h>
#include <cilk/reducer_opadd.h>

cilk::reducer_opadd<int> X;

void f() {
    X += 1;
}

int g() {
    cilk_spawn f();
    X += 2;
    cilk_sync;
    return X.get_value();
}

Variable X is declared as a hyperobject for doing addition reduction over type int.    Here is what happens when function g() is called and there is an idle thread that successfully steals.

1. The cilk_spawn causes control flow to fork into two strands of execution.  One strand calls f(), which executes X+=1.
2. The other strand is a continuation of the caller's execution.  The idle thread may steal the continuation and executes "X+=2".  If stealing does not occur, the strand executes after f() returns.
3. Execution waits at the cilk_sync until both strands complete. 
4. The value of X is returned. 

As a practical matter, stealing is unlikely in this example because f() executes so quickly that the original thread will get to execution of the continuation before a thief can grab it.  But for exposition's sake, assume that += is slow. 

If X were an ordinary int, having two strands concurrently update X would be unsafe, because one of the updates might stomp on the other.  Declaring it as a hyperobject avoids the problem.  When the thief operates on X, it gets a gets a fresh view, initialized to 0, the identity element for addition.   The cilk_sync causes the views to be merged into a single view.  The declaration of X implies merging by addition, so the net effect of calling g() is X+=3.  

If the continuation is not stolen, a fresh view is not created.  The X+=1 happens first, followed by X+=2, both on the same view.  Thus the next effect of g() is still X+=3.

The reduction operation should be associative, or as in the case of floating-point addition, practically associative for given circumstances.  But it need not be commutative.   When two views merge, the reduction operation is always applied such that the left operand is the view for the spawned routine and the right operand is the view for the stolen continuation.  

Now about detecting steals.  The idea is to detect when a view is fresh.  I'll use a global boolean hyperobject "Seen" for this purpose.   I'll use "Seen==false" to indicate that a view is fresh.  This convention is simplifies the code by exploiting default initialization of new views, so I do not have to explicitly specify their value.  

The domain of the "reduction" is boolean values.  The reduction operation is "x op y → x".  It's associative but not commutative.  A hyperobject with this reduction operation is called a holder, because it holds the left value.   The right value is irrelevant because it corresponds to the view created by a thief, and inspected by that thief.    (Exercise for the mathematically inclined: do left and right identity values exist for this operation?  What are they?)

Here is a complete example of using a holder to detect steals.  You can compile and run with Intel(R) Cilk Plus compiler.    

#include <cilk/cilk.h>
#include <cilk/reducer.h>

template<typename U>
class Holder {
    struct Monoid : cilk::monoid_base<U> {
        static void reduce(U *left, U *right) {}
    };

    cilk::reducer<Monoid> impl;
public:
     inline U& get_view() {
        return impl.view();
    }
};

Holder<bool> Seen;

#include <cstdio>

int main() {
    Seen.get_view() = true;
    cilk_for( int i=0; i<100000000; ++i ) {
        bool& x = Seen.get_view();
        if( !x ) {
            std::printf("Iteration %d was stolen\n",i);
            x = true;  // Must not forget this part.
        }
    }
    return 0;
}

 Here is an explanation of the program's parts:

• Template Holder<U> defines a holder for views of type U, using the template cilk::monoid_base defined in <cilk/reducer.h>.  
• Template class cilk::reducer<Monoid> requires that signature Monoid::reduce compute "*left = *left op *right".  The implementation is trivial for the holder reduction operation  "x op y → x".
• The views conceptually live in Holder::impl.  A strand invokes impl.view() to get a reference to its view.
• Seen is declared as a Holder<bool>.  The views of the bool live in Seen.impl.  The initial view is default initialized to false.
• Function main marks the initial view as seen.
• Function main executes a cilk_for loop, which parcels out chunks of iterations as work.
• Each iteration inspects its view of Seen.  If the view is false, then it is a freshly created view, which indicates that the chunk was stolen.  The view is  marked as seen after the theft is reported.

The code abuses hyperobjects in the sense that its visible behavior depends on whether steals happen or not.  Not all uses of holders are abusive.  Consider a scratchpad variable that is used for temporary storage, but its final value does not matter.  Changing the variable to a holder enables a Cilk program to safely operate on it, without any locks, because each thread will get its own view as necessary.  Furthermore,  each view is operated on in the left-to-right order of the original program.  For some applications, that's a valuable property that enables maintaining complex state in the scratchpad, which is not necessarily a practical thing to do with thread-local storage.

Footnote: Intel and Cilk are trademarks of Intel Corporation in the U.S. and/or other countries.

Background

As outlined in the first blog, fork-join and SIMD parallelism can be combined to solve a computational problem:

• fork-join parallelism can specify which subproblems can be solved in parallel by different hardware threads.
• SIMD parallelism can be applied to each subproblem, which helps the compiler exploit SIMD instructions.

In fork-join parallelism, control flow forks into separate flows, and later the flows join back together.  Recursive fork-join can be particularly effective.  For example, 10 levels of two-way splits creates 1024-way potential parallelism.  1024 separate hardware threads would be inefficient.  But systems like Cilk (and TBB)  are careful to turn potential parallelism into actual parallelism only when necessary.  Indeed with these systems you should specify as much fork-join parallelism as you can, and let the system determine when, where, and how much to exploit.

An important point is that the subproblems should fit in cache, otherwise multiple threads can easily run up against memory bandwidth limitations.  Sometimes it is hard to determine what a cache-sized subproblem is.  When in doubt, go as small as practical.  The subproblems are too small if the overhead of scheduling a chunk  becomes significant compared to the work for solving the subproblem serially.  Because Cilk Plus is tightly integrated into the compiler, and has highly structured parallelism, it tends to have lower chopping overhead than systems like TBB that are less structured and supplied as  libraries.  So you can often afford to chop into finer subproblems than you might with a system such as TBB.  [If you are interested in a deep comparison of Cilk vs. TBB, let me know, and I'll post some blogs on the subject.]

A particularly effective form of chopping is exhibited by cache oblivious algorithms.   These optimize for all levels of cache  that might exist (including virtual memory!), oddly enough without knowing anything about the levels.  See the link for more information.

Quick Introduction to Cilk Notation

There are two ways to express fork-join parallelism in Cilk:

• cilk_spawn / cilk_sync
• cilk_for

For example, the statement cilk_spawn f(); asynchronously calls a function f.  Here asynchronous means that thethe caller keeps going without waiting for the callee to return.   This is a neat way to express parallelism because most of what you learned about subroutine calls still applies.  The actual arguments are evaluated, bound to formal arguments, and the callee's body is invoked.    The only thing that changes is that the caller continues to execute after the callee is entered, if there is a hardware thread available to continue execution of the caller.

Eventually a caller must wait on its callees.  To do so, the caller invokes cilk_sync.  For example, the following code lets f(x), g(y), and h(z) run in parallel if there are sufficient hardware resources. It waits for them to complete before printing their results.

a = cilk_spawn f(x);
b = cilk_spawn g(y);
c = h(z);
cilk_sync;
cout << a << b << c << "\n";

Here is a picture that shows the execution order.

The last call is not spawned as a matter of good Cilk style.  It could be spawned, but doing so  is considered poor style, because there is no other work to do between that call and the cilk_sync.  Doing nothing in parallel with doing something is pointless overhead.

The semantics of cilk_sync is that it waits for all cilk_spawn calls that were issued by the current Cilk block. A Cilk block is the body of a function, or the body of a cilk_for loop. Complex control flow is allowed between cilk_spawn and cilk_sync.  For example, this code:

for( list::iterator i=x.begin(); i!=x.end(); ++x )
    cilk_spawn f(*i);
cilk_sync;

walks a linked list x sequentially and issues calls f(*i) that run concurrently.  Execution waits for all of them to complete at the cilk_sync.

There is always an implicit cilk_sync when a routine returns.  Hence a callee cannot accidentally leave "dangling tasks" running after it returns. This feature makes it much easier to reason about Cilk parallelism than it is to reason about raw threads.

The callee can be function object (functor) too.  Since a C++1x lambda expression returns a function object, you can also spawn a block of work like this:

cilk_spawn [&]{for( int i=0; i<5; ++i ) cout<<"quack ";} ();
other_work(); // Do other work while quacking.
cilk_sync;

In the example, the lambda expression [&]{...} returns a function object.  The  cilk_spawn...() causes it to execute asynchronously.  Do not forget the trailing parentheses, or the compiler will balk.

The notation cilk_for is like a C/C++ for, except that iterations run in parallel if resources permit.   For example:

cilk_for( vector::iterator i=x.begin(); i!=x.end(); ++x )
    f(*i);

allows, but does not mandate, each loop iteration to run in parallel.  A cilk_for loop is usually more efficient than a serial for loop wrapped around cilk_spawn requests.  The reason is that a cilk_for can distribute the work across processors in parallel, and much more efficiently, than the combination of for and cilk_spawn.  However, a cilk_for is more limited than a plain for.  The iteration variable must be of an integral type or a random access iterator, and the cilk_for must fit a certain pattern, which permits the number of iterations to be computed before the loop iterations commence.  See the documentation here for details on the limitations.

Another key feature of Intel(R) Cilk Plus are hyperobjects.  They deal the "join" part of fork-join parallelism when there is data to be joined, such as in reductions.  Hyperobjects are a topic for another day.

Example

If you've read my other recent blogs, you know I'll use  Seismic Duck as the example.  Two parts of Seismic Duck lend themselves to fork-join parallelism. The first is the three physical models that can be updated in parallel:

• wavefield propagation
• seismogram
• reservoir

This parallelism can be expressed with cilk_spawn and cilk_sync.  Here is the code from the actual source:

WavefieldFunctor wf(subsurface, pausedRequest);
cilk_spawn wf();
SeismogramFunctor sf(seismogramClip, pausedRequest&NimbleDraw);
cilk_spawn sf();
ReservoirFunctor rf(pausedRequest);
rf();
cilk_sync;

Two of the functors turn out to be just wrappers around function calls.  They were wrapped because I was using TBB.  I really don't need the wrappers  in Cilk.  For example, SeismogramFunctor looks like:

class SeismogramFunctor {
    const NimbleRequest request;
    NimblePixMap seismogramClip;
public:
    SeismogramFunctor( const NimblePixMap& seismogramClip_, NimbleRequest request_ ) :
        seismogramClip(seismogramClip_),
        request(request_)
     {}
     void operator()() const {
         SeismogramUpdateDraw( seismogramClip, request, TheColorFunc, IsAutoGainOn );
     }
};

so instead of writing that class definition and the lines:

    SeismogramFunctor sf(seismogramClip, pausedRequest&NimbleDraw);
    cilk_spawn sf();

I could have discarded the wrapping and just written:

    cilk_spawn SeismogramUpdateDraw( seismogramClip, request, TheColorFunc, IsAutoGainOn );

That's short and sweet.

The second part that can be profitably parallelized is wavefield propagation, by using geometric decomposition and the ghost cell pattern, as described in another blog.  There are two phases:

• Exchange boundary information between panels.
• Update each panel independently.

The second phase dominates execution time, so that's the part I'll parallelize.  The parallel code is:

cilk_for( int p=0; p<NumPanel; ++p ) {
    if( request&NimbleUpdate )
        WavefieldUpdatePanel( p );
    if( request&NimbleDraw )
        WavefieldDrawPanel( p, map );
}

All I had to do was change the keyword for to cilk_for.  As I mentioned earlier, to fully utilize the processor, we also need SIMD parallelism.  The panel updates are further divided into sequences of tile updates that improve cache reuse.  Part 1 explained how a tile updatecan be expressed concisely with Array Notation .

Serial Semantics

It's no accident that the parallel code is almost identical to the original serial code.  Cilk is essentially annotations on a serial program that say where parallelism is permitted.  In fact, your code can still compile with non-Cilk compilers, albeit without the parallelism. Just use the following defines to undo the annotations:

#define cilk_spawn
#define cilk_sync
#define cilk_for for

Code compiled this way behaves exactly the way the parallel program behaves if limited to a single hardware thread.  I hope that eventually all C/C++ compilers recognize Cilk notation so that the #define trick will not be necessary and Cilk parallelism becomes ubiquitous.

Summary

Cilk notation makes fork-join parallelism easy.  Array Notation makes SIMD parallelism easy.  Intel(R) Cilk Plus provides both.  Combined, they are a powerful pair of tools for squeezing high performance from modern microprocessors.

Background

A while back I wrote about the Three Layer Cake pattern for parallel programming.  The pattern is a way of organizing programs to fully exploit modern  multi-core chips.   Two of the layers are:

• fork-join: harnesses multiple hardware threads.
• SIMD: harnesses SIMD instructions.

The compiler in Intel Parallel Composer 2011 extends C++ to directly support these two layers.  The extensions are called Intel(R) Cilk Plus.  They are:

• Cilk notation for specifying fork-join parallelism.
• Array notation for specifying SIMD parallelism.

This blog introduces the array notation, with a Seismic Duck kernel as the example.  I'll introduce Cilk notation in another blog.  The two notations are independent.  Indeed, the array notation is valuable with other threading packages too, such as Threading Building Blocks, or just for writing faster serial code.

Quick Introduction to Array Notation

The array notation extension is reminiscent of APL and Fortran-90 style array expressions.   The expression:

a[index:count]

denotes an array section starting at index with count elements.  Scalar operations can be used on conformable array sections in an intuitive manner.   Operations between scalars and array sections work too; scalar extende in the obvious way (like in APL or Fortran 90).  Examples:  

    z[i:n] = x[i:n];      // Copies x[i..i+n-1] to x[i..i+n-1].
    z[i:n] = 2*x[i+1:n];  // Sets z[i..i+n-1] to twice the corresponding elements in x[i+1..i+n].
    u[i:m][j:n] += 1;     // Increments elements of two-dimensional mxn array section with upper left corner [i][j].

Section notation also permits expressions of the form array[index:count:stride], reductions, and shorthands that I will not into here.   I'm presenting just enough to pique your interest.  To learn more about it, follow this link to the compiler documentation.   

Example

I've described in other blogs how seismic wave propagation in Seismic Duck depends on updating a "tile", a small subarray that fits in cache.  Here is the scalar code that dominates execution time.  It updates a tile with uniform A and B coefficients:

    float a = 2*A[iFirst][jFirst];
    float b = B[iFirst][jFirst];
    for( int i=iFirst; i<iLast; ++i ) {
        for( int j=jFirst; j<jLast; ++j ) {
            Vx[i][j] += a*(U[i][j+1]-U[i][j]);
            Vy[i][j] += a*(U[i+1][j]-U[i][j]);
            U[i][j] += b*((Vx[i][j]-Vx[i][j-1])+(Vy[i][j]-Vy[i-1][j]));
        }
}

To improve the speed on compilers that did not automatically generate SIMD code from the scalar loops, I wrote the key loops with SSE intrinsics, so that calculations are done four wide instead of one at a time.  The resulting code looks like this:

    #define CAST(x) (*(__m128*)&(x))        /* for aligned load or store */
    #define LOAD(x) _mm_loadu_ps(&(x))      /* for unaligned load */
    #define ADD _mm_add_ps
    #define MUL _mm_mul_ps
    #define SUB _mm_sub_ps
    ...
    __m128 a = CAST(A[iFirst][jFirst]);
    a = ADD(a,a);
    __m128 b = CAST(B[iFirst][jFirst]);
    for( int i=iFirst; i<iLast; ++i ) {
        for( int j=jFirst; j<jLast; j+=4 ) {
            __m128 u = CAST(U[i][j]);
            CAST(Vx[i][j]) = ADD(CAST(Vx[i][j]),MUL(a,SUB(LOAD(U[i][j+1]),u)));
            CAST(Vy[i][j]) = ADD(CAST(Vy[i][j]),MUL(a,SUB(CAST(U[i+1][j]),u)));
            CAST(U[i][j]) = ADD(u,MUL(b,ADD(SUB(CAST(Vx[i][j]),LOAD(Vx[i][j-1])),SUB(CAST(Vy[i][j]),CAST(Vy[i-1][j])))));
        }
    }

The downside of the change is obvious - it's hard to read. And this was a simple case because logic elsewhere guarantees that jLast-jFirst is a multiple of 4.  Otherwise, dealing with the extra iterations would have further obfuscated the code.

For this particular example, explicit SSE intrinsics are not actually necessary with a compiler that automatically vectorizes (convert to SIMD instructions).  Indeed, recent compilers that I tried seem to be able to do so.  (Though one older compiler from 2008 did not.)   But I was careful to cater to the optimizer.  I declared the arrays Vx, Vy, and U as static file-scope arrays in the source code, not pointers.  That's not trendy OO programming, but it lets the compiler easily prove absence of aliasing, and thus absence of loop carried dependences that could thwart vectorization.   It's not always practical to cater this way to the optimizer.  Furthermore, array notation has its own elegance.  So I'll use the kernel as a running example anyway. 

The array notation in Intel(R) Cilk Plus lets me state my intent ("SIMD parallelism!") to the compiler more bluntly.  Below is an array notation version of the example:

   int i = iFirst;
   int j = jFirst;
   size_t m = iLast-iFirst;
   size_t n = jLast-jFirst;
   float a = 2*A[i][j];
   float b = B[i][j];
   Vx[i:m][j:n] += a*(U[i:m][j+1:n]-U[i:m][j:n]);
   Vy[i:m][j:n] += a*(U[i+1:m][j:n]-U[i:m][j:n]);
   U[i:m][j:n] += b*((Vx[i:m][j:n]-Vx[i:m][j-1:n])+(Vy[i:m][j:n]-Vy[i-1:m][j:n]));

Compare the last three lines with the forall loops from which the code was derived in another blog:

    forall i, j {
        Vx[i][j] += (A[i][j+1]+A[i][j])*(U[i][j+1]-U[i][j]);
        Vy[i][j] += (A[i+1][j]+A[i][j])*(U[i+1][j]-U[i][j]);
    }
    forall i, j {
        U[i][j] += B[i][j]*((Vx[i][j]-Vx[i][j-1]) + (Vy[i][j]-Vy[i-1][j]));
    }

The array notation has let me clearly convey the parallel nature of the updates.  I had to add the setup of i, j, m, n.  But that's an accident of history.  Elsewhere the code computes { iFirst, iLast, jFirst, jLast} from the equivalent of {i, j, m, n} because the former simplified writing the C++ for loops.  If I adapt the rest of the code to use array notation, then { iFirst, iLast, jFirst, jLast} will disappear and {i,j,m,n} will be setup in their place.

Of course I'm still depending on a clever optimizer to eliminate the temporary subarrays.  For example, the subexpression:

    a*(U[i:m][j+1:n]-U[i:m][j:n]);

conceptually generates two temporary array sections, for the results of - and *.   In practice, the Intel compiler is good at eliminating those temporaries.   (It's had years of practice doing so for Fortran 90.)  But even if some temporary sections remained, the parallelism is still clear.  The compiler does not have to deduce parallelism from dependence analysis of serial for loops. 

Concise notation is nice, but how about the performance?  When compiled by the Intel compiler and run on the Core-2 Quad system in my office, the array notation variant performed faster than my hand-coded SSE.   [Your mileage may vary.  See optimization disclaimer here.]  I dug through the object code to figure out why.  It turns out that updating Vx, Vy, and U in separate loops does better than with a single loop.   I found out that the hand-coded SSE does as well if changed to use separate loops to update the three arrays.  Anyway, I'm happy that the array notation matches the best that I can do by hand for this example.

Summary

The array notation is a concise way to express SIMD parallelism.  I'm hoping it catches on with other compilers.  In another blog I'll introduce the application of Cilk fork-join parallelism to Seismic Duck.

• My blogs on parallelizing it with SSE and TBB omit details of interest.  The blogs chiefly concern the seismic wave propagation code in Source/Wavefield.cpp .
• Games about reflection seismology are not runaway best sellers.
• It's limited to Windows.  I'd like to find volunteers to port it to other platforms.   Mac OS is of particular interest, since it is common in educational settings.   The OS-specific parts are about 600 lines of C++.

The code is my own hobby, not Intel's.   As such, comments are  sparse.  I'll expand them as questions arise.  If you are interested in porting it to new platforms, please contact me and I'll help as best I can.

Vectorization

I'll describe vectorization first.  The kernel of interest is:

        for( i=start[k]; i<finish[k]; ++i )     // Loop over rows in chunk k
            for( j=1; j<n; ++j ) {
                Vx[i][j] += (A[i][j+1]+A[i][j])*(U[i][j+1]-U[i][j]);
                Vy[i][j] += (A[i+1][j]+A[i][j])*(U[i+1][j]-U[i][j]);
                U[i][j] += B[i][j]*((Vx[i][j]-Vx[i][j-1]) + (Vy[i][j]-Vy[i-1][j]));
            }

A Core-2 core can generally operate on 4-element vectors just as fast as it can operate on scalars.  Thus  scalar code wastes 3/4 of the machine's capability.  The kernel is trivial to vectorize.  Let X[i:4] denote a four-element SSE vector, starting at subscript i.  In other words, X[i:4] is the quadruple (X[i], X[i+1], X[i+2], X[i+3]). To vectorizing this kernel:

• Replace the scalar operations with SSE vector operations
• adjusting the stride of the j loop index to loop over SSE vectors instead of scalars

The vectorized code looks like this conceptually:

        for( i=start[k]; i<finish[k]; ++i )     // Loop over rows in chunk k
            for( j=1; j<n; j+=4 ) {
                Vx[i][j:4] += (A[i][j+1:4]+A[i][j:4])*(U[i][j+1:4]-U[i][j:4]);
                Vy[i][j:4] += (A[i+1][j:4]+A[i][j:4])*(U[i+1][j:4]-U[i][j:4]);
                U[i][j:4] += B[i][j:4]*((Vx[i][j:4]-Vx[i][j-1:4]) + (Vy[i][j:4]-Vy[i-1][j:4]));
            }

The vectorization is legal because afterwards, the updates of Vx and Vy still use the old values of U, and the updates of U still use the new values of V. Note that allowing multiple iterations to run concurrently using threading would not be safe, as explained in part 2. The loop thus demonstrates the the point that not all vectorizable loops can be multithreaded, at least without further transformations, e.g. the ghost cell pattern covered in part 2.

The beta version of the Intel 12.0 C++ compiler actually supports the [j:4] array syntax. It's really convenient to use. Alas I was developing Seismic Duck without that support, so I resorted to SSE intrinsics and macros. The macros look like this:

        #define CAST(x) (*(__m128*)&(x))    /* for aligned load or store */
        #define LOAD(x) _mm_loadu_ps(&(x))    /* for unaligned load */
        #define ADD _mm_add_ps
        #define MUL _mm_mul_ps
	#define SUB _mm_sub_ps

With these macros the loop body looks like:

        __m128 u = CAST(U[i][j]);
        __m128 a = CAST(A[i][j]);
        CAST(Vx[i][j]) = ADD(CAST(Vx[i][j]),MUL(ADD(LOAD(A[i][j+1]),a),SUB(LOAD(U[i][j+1]),u)));
        CAST(Vy[i][j]) = ADD(CAST(Vy[i][j]),MUL(ADD(CAST(A[i+1][j]),a),SUB(CAST(U[i+1][j]),u)));
	CAST(U[i][j]) = ADD(u,MUL(CAST(B[i][j]),ADD(SUB(CAST(Vx[i][j]),LOAD(Vx[i][j-1])),SUB(CAST(Vy[i][j]),CAST(Vy[i-1][j])))));

It's efficient ugly non-portable vector code.  I'm hoping that other compilers adopt the array syntax so I can write efficient beautiful portable vector code.

Tiling

Vectorization alone does not improve performance of the kernel.   It slams it harder against the "memory wall".   The "memory wall" refers to the limit on memory bandwidth.  Reducing consumption of memory bandwidth becomes critical.  Parts 1 and 2 discussed the importance of the ratio C (Compute density).  It is defined as the ratio of the number of floating-point additions per memory reference.  Now I'll describe how raise the value of C multifold with tiling.

The kernel as shown so far updates the wavefield for a single time step. Now it's time to bring in more context. The kernel occurs inside a loop that steps over time, conceptually like this:

    for( t=0; t<T; ++t ) {  // T = number of time steps per video frame (typically 3)
        replicate borders
        for each chunk k in parallel {
            for( i=start[k]; i<finish[k]; ++i ) // Loop over rows in chunk k
                for( j=1; j<n; j+=4 ) {
                    Vx[i][j:4] += (A[i][j+1:4]+A[i][j:4])*(U[i][j+1:4]-U[i][j:4]);
                    Vy[i][j:4] += (A[i+1][j:4]+A[i][j:4])*(U[i+1][j:4]-U[i][j:4]);
                    U[i][j:4] += B[i][j:4]*((Vx[i][j:4]-Vx[i][j-1:4]) + (Vy[i][j:4]-Vy[i-1][j:4]));
                }

With the loop structure above, each iteration of the t loop has to reload Vx, Vy, and U from memory.  If the t loop could be changed from being the outermost loop to being the innermost loop, the reloads could be avoided, because each iteration would reuse values cached by the previous iteration. But naively permuting the loops delivers wrong answers, because updates would use incorrect values.

Ignore the "for each chunk" loop for now.  I'll come back to that in the next section.  For now, consider the other three loops. They are an abstract loop over points in space-time with coordinates (t,i,j).  The constraints are that point (t,i,j) must be updated:

• after update of point (t-1,i,j) (required to preserve dependencies across time)
• after update of points (t,i-1,j) and (t,i,j-1) (required to update U correctly)
• before update of point (t,i+1,j) and (t,i,j+1) (required to update Vx and Vy correctly)

The tiling will update the points in groups. Each group is a rectangular subset of points, called a tile.  A tile is updated using two nested loops as shown in the vectorized code, but i and j range over the tile instead of the entire spatial grid. An additional outer loop sequences the tiles.

Picture the kernel as a problem in tiling a floor. The floor runs along the i and j axes.  The t axis is vertical to the floor.  The floor must be tiled T levels deep for each video frame.  T is typically 3 in Seismic Duck. Each tile of size 1×7×112.  That is, each tile is:

• 1 unit thick along the t axis,
• 7 units thick along the i axis, and
• 112 units along the j axis.

Why such a long skinny tile and where did the 112 come from? The answer is that the tiles are actually square, when measured in cache lines. It's cache lines that count, because they are the quantum of information transferred between memory and the CPU. A cache line is typically 64 bytes on my intended target hardware, and I'm using single-precision floating-point.  So that's 16 floats per line.  The 7×7 dimension in cache lines works out to be about the right size.  I chose the size by a back of the envelope calculation (based on the size of the L1 cache) and then did some experiments.

Tiles can be cut into smaller tiles if necessary, particularly when working around the edges of the grid.  Part 4 will way more about this.  The previous constraints on the order of updating points apply similarly to tiles. A tile can be laid at level t only if:

• all the area under it is tiled to level t-1,
• its north and west sides will not be exposed,
• its south and east sides will be exposed.

Another way to word the constraints is from the viewpoint of an ant crawling over the tiles:

• No overhanging tiles allowed.
• If an ant crawls north or west, it must never go down.
• If an ant crawls south or east, it must never go up.

Below is a picture showing both legal and illegal partial tilings. The three levels of tiles are distinguished by different colors.

The red X marks tiles that violate one or more of the constraints.

There is one more thing to model: the cache hardware. Laying down a tile corresponds to performing a calculation on data that must be in cache.  Each tile is a 7x7 grid of cache lines.  Call a line hot if it is in cache, and cold if it is out of cache.  If a cache line is not touched, it cools until it becomes cold. A cache line can be reheated (brought back into cache), but that takes precious time. These effects add a constraint to the tiling exercise, because the computation of points in a tile depend on nearby points in space. When laying a tile, the area under and close to the new tile must be hot.

Laying a complete layer before doing the next layer is inefficient because each time a tile is laid, the area until it has grown cold.  It's far better to lay a tile on top of a hot tile, ideally the one laid immediately earlier.  A good pattern is to lay each tile almost on top of the previous tile, but shifted north and west one unit to meet the aforementioned "ant constraints", until the stack of offset tiles is T levels deep. Then lay another stack immediately east of this stack. Once a row of stacks is completed, do the next row.

The pattern looks tricky around the edges of the floor, but there is an easy way to deal with that. Imagine that we do not care if a tile goes beyond the floor boundary, and the overhang constraint does not apply to beyond the floor. Then it's just a matter of starting the pattern beyond the floor, and cutting off portions of a tile that extend beyond the floor.

Below is an animation of the tiling sequence for a small 23x16 floor with 7x7 tiles.

Note: I wrote the animation by hand using GIMP, frame by frame. I now greatly appreciate the effort artists put into hand-drawn animations!

With the typical depth of 3, the tiling pattern almost triples the compute density.  I say "almost" because I may not be getting perfect reuse around the edges.   But it's close enough.   Some programs sequence the stacks more elaborately, such as zig-zagging diagonally or recursively, in order to exploit multiple levels of cache. The simple pattern I used works well for Seismic Duck because the outer level cache (e.g. L2 cache on a my system) is big enough to hold the cache lines that are reused between rows of stacks. T

The final loop structure looks like:

    for each video frame do {
        replicate chunk borders widely // See next section
	for each chunk k in parallel {
	    for each tile z in chunk k, in tiling sequence order, do
		for each i in z do
		   for each j in z do {
                       Vx[i][j:4] += (A[i][j+1:4]+A[i][j:4])*(U[i][j+1:4]-U[i][j:4]);
                       Vy[i][j:4] += (A[i+1][j:4]+A[i][j:4])*(U[i+1][j:4]-U[i][j:4]);
                       U[i][j:4] += B[i][j:4]*((Vx[i][j:4]-Vx[i][j-1:4]) + (Vy[i][j:4]-Vy[i-1][j:4]));
		    }

Part 4 will say more about implementing the tiling sequence.

Wide Halo

Note that I added the word "widely" to the replication step. Because the tiling sequence advances a chunk T timesteps instead of one, it is not enough to copy just the immediate border points. Instead, a T-point wide border has to be copied, and the space to be tiled is not a rectangular prism in (t,i,j) space, but instead is a frustum with a rectangular base in the (0,*,*) plane, and walls that slope 45 degrees. The top surface of the frustum in the (T-1,*,*) plane.  The top surface is the chunk's contribution to the final computation. The bases overlap by a region T points wide. The overlapping portions of the frustums correspond to redundant calculations.  See the section titled "Wide Halo" of the Ghost Cell Pattern paper for more on this technique.

Summary of Vectorization and Cache Optimizations

Vectorization quadruples the calculations per instruction, making memory bandwidth the bottleneck. Tiling in space-time reduces consumption of memory bandwidth. Doing so requires the wide halo pattern.

The tiling involves some complex bookkeeping.  Boundary condition physics makes it even more complicated.  Part 4 will describe how I dealt with the bookkeeping without going mad.

The big advantage of learning parallel patterns is twofold.  First, you learn a way to solve a class of parallel programming problems.  Second, you gain a concise way to describe the solution, because the pattern has a name.   Like a standard recipe, you can point someone unfamiliar with the pattern to paper describing the pattern, instead of having to explain it from scratch each time.

Part 1 described the update calculations as arranged in the Odd-Even pattern:

    forall i, j {
        Vx[i][j] += (A[i][j+1]+A[i][j])*(U[i][j+1]-U[i][j]);
        Vy[i][j] += (A[i+1][j]+A[i][j])*(U[i+1][j]-U[i][j]);
    }
    forall i, j {
        U[i][j] += B[i][j]*((Vx[i][j]-Vx[i][j-1]) + (Vy[i][j]-Vy[i-1][j]));
    }

Part 1 described how this form is trivial to parallelize, but suffers from doing little computation per memory reference. It defined the ratio C (Compute density) to be the number of floating-point additions per memory reference. Using the Odd-Even pattern for the wave simulation yields C=10/11=.91; the hardware can deliver C=12. In other words, memory bandwidth limits the floating-point unit to about 1/13th of its capability. [Whence the 13th?  12/(10/11)=13.2]

Updating Vx, Vy, and U in a single sweep improves C. Here is a breakdown of the work per grid point for the single sweep method:

• Single sweep (update Vx, Vy, and U):
  • 8 memory references (read A, B, U, Vx, Vy; write Vx, Vy, U).
  • 10 floating-point adds
  • 3 floating-point multiplications

Now C=10/8=1.25. That’s about a 37% improvement over the C=.91 value in the original code. Part 3 will show how to triple that value, to reach C=3.75 and even raise it higher. However, the improvement in C complicates parallelism. The single-sweep version fuses the original loops to look like this:

    for( i=1; i<m; ++i )
      for( j=1; j<n; ++j ) {
        Vx[i][j] += (A[i][j+1]+A[i][j])*(U[i][j+1]-U[i][j]);
        Vy[i][j] += (A[i+1][j]+A[i][j])*(U[i+1][j]-U[i][j]);
        U[i][j] += B[i][j]*((Vx[i][j]-Vx[i][j-1]) + (Vy[i][j]-Vy[i-1][j]));
      }

The fused code has a sequential loop nest, because it must preserve the following constraints:

• The update of Vx[i][j] must use the value of U[i][j+1] from the previous sweep.
• The update of Vy[i][j] must use the value of U[i+1][j] from the previous sweep.
• The update of U[i][j] must use the values of Vx[i][j-1] and Vy[i-1][j] from the current sweep.

Treating the grids as having map coordinates, a grid point must be updated after the points north and west of it are updated, but before the points south and east of it are updated.

One way to parallelize the loop nest is the Wavefront Pattern . In that pattern, execution sweeps diagonally from the northwest to southeast corner. But that pattern has relatively high synchronization costs. Furthermore, in this context, it has poor cache locality because it would tend to schedule adjacent grid points on different processors. Some of these inefficiencies can be ameliorated by aggregating grid points into chunks. But nonetheless it is less attractive than the following alternative.

The alternative is geometric decomposition and the Ghost Cell Pattern. Here is a picture of the geometric decomposition in Seismic Duck:

The picture is approximately to scale. Part 3 explains the rationale for choosing such skinny chunks. (Hint: they are not skinny when you use the right measurement units.) Each chunk can be updated independently by a different thread, except around its border. To see the exception, consider the interaction of chunk 0 and chunk 1. Let i₀ be the index of the last row of chunk 0 and let i₁ be the index of the first row of chunk 1.

1. The update of Vy[i₀][j] must use the value of U[i₁][j] from the previous sweep.
2. The update of U[i₁][j] must use the value Vy[i₀][j] from the current sweep.

The ghost cell pattern enables the chunks to be updated in parallel. Each chunk becomes a separate grid with an extra row of grid points added above and below it. These extra rows replicate information from the neighboring chunks so that each chunk has a copy of the grid points just beyond it. The copy enables a thread working on chunk 0 can get the value of U[i₁][j] from the previous sweep even if the thread working on chunk 1 updates its copy of U[i₁][j]. Likewise the thread working on chunk 1 can update its copy of Vy[i₀][j] without waiting for the thread working on chunk 0 to update it.

The update logic ends up looking like this:

    // Replicate borders
    for( k=0; k<number_of_chunks-1; ++k ) {
        copy bottom border of chunk k to top of chunk k+1
        copy top border of chunk k+1 to bottom of chunk k
    }
    // Update chunks in parallel
    forall k in 0..number_of_chunks-1 {
        for( i=start[k]; i<finish[k]; ++i ) // Loop over rows in chunk k
            for( j=1; j<n; ++j ) {
                Vx[i][j] += (A[i][j+1]+A[i][j])*(U[i][j+1]-U[i][j]);
                Vy[i][j] += (A[i+1][j]+A[i][j])*(U[i+1][j]-U[i][j]);
                U[i][j] += B[i][j]*((Vx[i][j]-Vx[i][j-1]) + (Vy[i][j]-Vy[i-1][j]));
            }
    }

For more insight into this pattern, see the Ghost Cell Pattern paper.  It’s a polished paper on an important pattern for grid-based simulations.

The final part 3 of these blogs will cover the cache optimization patterns, which are critical because memory bandwidth, not computation, is the bottleneck.

Breaking Changes

The breaking changes for TBB 3.0 are that the following methods of class task are changed to static methods:

• spawn
• destroy
• allocate_additional_child_of

The changes are unlikely to break existing code, because these methods are almost always used in expressions such as “x.spawn(y)”, where x and y are expressions.  Though spawn is now a static method, the expressions x and y are still evaluated.  The only difference is that the value of expression x is no longer used by method spawn.

A source-level break occurs in code that takes the address of one of the changed methods.  For example:

     typedef void (task::*ptr)(task&);

     ptr p = &task::spawn;

Such code will now elicit an error message from the compiler. The edit below shows how to fix it:

     typedef void (task::*ptr)(task&);

     ptr p = &task::spawn;

Evolution motivated the change. The expression “x.spawn(y)” originally meant that task x spawns task y.  The task interface evolved in a way that made “x” unnecessary.  Starting from scratch, we would probably define “y.spawn()” to mean to “spawn y”.  Introducing that form now while retaining “x.spawn(y)” would make TBB confusing to existing users, and we did not want to break code using the old form.  

Making spawn static seems like a good compromise.  Old code still works and new code does not need to supply the superfluous x. The only drawback is that code sometimes must supply a class qualification, as in “task::spawn(y)”.  Our examples in the Tutorial needed no change on this point – the call was always in a context where the task:: prefix was implied.   In another blog, I’ll explain how we made old application binaries still work with the new library, even though the signatures have changed.  

The only other theoretically breaking change is that the functor parameters to “parallel_invoke” and “parallel_for_each” are now passed by const reference instead of by value.  We actually made this change in a TBB 2.2 update 2, and did so to be compatible with the corresponding PPL implementations. 

Deprecations

As a standards committee maven once said, deprecation is a “warning shot across the bow”.  It is a warning that a feature might disappear in the future.  So far nothing has disappeared in TBB, even 1.0 features.   I still recommend discontinuing use of deprecated features, because the features might disappear in the future.  A feature is usually deprecated because a better alternative exists. 

The features newly deprecated in TBB 3.0 are:

• task::recycle_to_reexecute.  A call “t-> recycle_to_reexecute();” can be replaced with the sequence:

            t->set_ref_count(1);
            t->recycle_as_safe_continuation();

• tbb::tbb_thread .  A use of tbb::tbb_thread can be replaced with TBB’s std::thread.

We did not remove the functionality of tbb_thread.  We renamed it std::thread, as in the C++0x draft.  It still has the same interface as the old tbb_thread.  It does not exactly match the C++0x specification, because doing so requires C++0x language features.  But it’s close enough for typical use cases.

We originally chose the name tbb::tbb_thread in TBB 2.1 to avoid name conflicts with vendor’s implementations of class std::thread, particularly in code that is not using namespace qualifiers.  It seemed like the right decision at the time.  But in TBB 3.0 we introduced many more C++0x features (e.g. condition_variable), where the tbb_ prefix was getting irksome. 

You get the TBB implementation of std::thread when you include tbb/compat/thread and TBB_IMPLEMENT_CPP0X=1. The default value of TBB_IMPLEMENT_CPP0X is 0 on platforms that implement std::thread. You can override our default definition to flip our implementation on or off.

Summary

Unless your code is using pointers to member functions of class tbb::task, your TBB 2.2 code should work just as well with TBB 3.0.

The game is Seismic Duck.  I recently rewrote it for Windows platforms.  It’s a complete rewrite of something I wrote for Macs in the mid 1990s.  It's a freeware game about reflection seismology, which is imaging underground structures by sending soundwaves into the ground and interpreting the echos. The program teaches the topic by letting the user experiment interactively. It sounds technical, but even children "get it" after playing with it a while.  I’m fond of writing games based on peculiar subjects. For example, Frequon Invaders is a Fourier transform game.  Reflection seismology is a subject I worked on at Shell in the early 1990s.

This series of blogs is about how and why Seismic Duck was parallelized with Intel(R) TBB differently than a related demo in the TBB distribution. Seismic Duck runs three independent core computations:

• Seismic wave propagation and rendering.
• Gas/oil/water flow through a reservoir and rendering.
• Seismogram rendering.

I run all three in parallel, using tbb::parallel_invoke, and vectorize the code where I can. At a hight level, it follows the Three Layer Cake pattern. (I'm one of the authors of that paper.) However, that level of parallelism was not enough to get a good animation speed on my machine.  The bottleneck was the wave propagation simulation, which is the focus of these blogs.

Some background on the numerical simulation of waves is necessary.  I used the popular "staggered grid" finite-difference time-domain (FDTD) method.  There are five 2D arrays, each representing a scalar field over a 2D grid.  Three of the arrays represent variables that step through time.  The other two represent rock properties.  The arrays are:

• B[i][j]: A constant array with vertex centered values. It’s constant in the sense that it changes only when the subsurface geology changes.
• A[i][j]: A constant array with vertex-centered values related to rock properties. The FDTD method actually requires the edge-centered values, but that would double the memory bandwidth for reading it, because for each interior grid point there is a horizontal edged and a vertical edge. As shown later, memory bandwidth, not computation, is the bottleneck. So to reduce memory references, edge-centered values are computed by averaging the two nearest vertex-centered values. Doing so saves memory references because there twice as many edges as vertices. The vertices actually store ½ their physical value, so that each edge value can be computed as the sum of its two endpoint values.
• U[i][j]: A variable array with vertex-centered values representing how much the rock is compressed at that point. It is variable in the sense that it changes for every time step.
• Vx[i][j], Vy[i][j]: Two variable arrays that represent the x and y component of the point’s velocity. This should not to be confused with the velocity of sound through the rock. These two grids are edge-center. Vx[i][j] represents a physical value between grid points [i][j] and [i][j+1]. Vy[i][j] represents a physical value between grid points points [i][j] and [i+1][j].

The picture below shows one square of the grid and how the array elements relate to the scalar fields. Notice how Vx and Vy are edge centered.

Array Vx and Vy are not only staggered halfway in space, they are staggered halfway in time.  At the beginning of a time step, Vx and Vy are a half a time step behind U.  The simulation advances Vx and Vy a full step, so that they become a half time step head of U. Then the simulation advances U a full time step. The algorithm is sometimes called “leap frog” because of how the arrays advance over each other in time.

The advantage of staggering and leap frogging is that it delivers results accurate to 2nd order for the cost of a 1st order approach.  The update operations are beautifully simple:

    forall i, j {
        Vx[i][j] += (A[i][j+1]+A[i][j])*(U[i][j+1]-U[i][j]);
        Vy[i][j] += (A[i+1][j]+A[i][j])*(U[i+1][j]-U[i][j]);
    }
    forall i, j {
        U[i][j] += B[i][j]*((Vx[i][j]-Vx[i][j-1]) + (Vy[i][j]-Vy[i-1][j]));
    }

The TBB demo "seismic" and Seismic Duck use two very different approaches to parallelizing these updates.  I wrote the original versions of both.   (Anton Malakhov has since made many improvements to my original TBB version.)  The reason for different approaches is different purposes:

• The TBB demo is supposed to show how to use a TBB feature ("parallel_for") and a basic parallel pattern. I kept it as simple as possible.
• Seismic Duck is written for high performance, at the expense of increased complexity. It uses several parallel patterns. It also has more ambitious numerical modeling, notably the use of perfectly matched layers to reduce artificial reflections from the simulation boundaries. (See the URL for a description of this fascinating technique that involves using complex-valued materials that do not exist in the real universe, but can exist in software!)

The pattern behind the TBB demo is "Odd-Even Communication Group" in time and a geometric decomposition pattern in space. The code flip-flops between updating the pair of arrays Vx and Vy , and updating array U. (Note to readers of that code: The variable names are S, T, and V instead of Vx, Vy, and U.) Each forall can be performed with a tbb::parallel_for loop. From a patterns perspective, this is a geometric decomposition pattern. These patterns make a good introduction to parallel programming, and require minimal changes for parallelization.

The big drawback of the Odd-Even pattern is memory bandwidth. When Seismic Duck is played on a typical 21” widescreen monitor, each grid has dimensions 531x1520. (You do not see the entire grid in the game – there is a hidden 16 pixel thick border for the perfectly matched layers.) That works out to about 16 MByte for all 5 grids. High end server have caches that large, but I’m targeting current desktop machines, which have caches in the 1-4 MByte range. Fortunately, a few consecutive rows of each array do fit in cache.

Thus using the Odd-Even pattern, each grid point is loaded once from main memory per time step. Here is a breakdown of work per grid point for each sweep:

• First sweep (update Vx and Vy):
  • 6 memory references (read A, U, Vx, Vy; write Vx, Vy).
  • 6 floating-point adds
  • 2 floating-point multiplications
• Second sweep (update U):
  • 5 memory references (read B, Vx, Vy, U; write U)
  • 4 floating-point adds
  • 1 floating-point multiplication

The multiplications are insignificant because the hardware can overlap them with the additions. The key consideration is the (6+4) floating-point additions per (6+5) memory references. For brevity, from now on I'll call this ratio C (for Compute density). A ratio of C=(6+4)/(6+5)=10/11≈0.91 is a serious bottleneck. My Core-2 Duo system can deliver 8 floating-point additions per clock (two cores each using 4-wide SIMD). But the memory bandwidth limits my system to 2 single-precision memory references every 3 clocks. (I used the STREAM benchmark to determine this.) So C for the hardware is 8/(2/3) = 12.

To summarize, C=12 for the hardware but C≈0.91 for the Odd-Even code. Thus the odd-even version delivers a small fraction of my machine’s theoretical peak floating-point performance. Phrased another way: forget the floating-point, it's the memory references that matter. Another approach was needed. Part 2 shows how I raised C to 1.25 and multi-threaded the wave simulation using TBB. Part 3 describes how I raised C much higher and vectorized the code.

I implemented a subset of task_group in TBB as a header tbb/task_group.h: 37 lines of C++ and 5 preprocessor lines.   It's a small subset.

• It does not support task_handle. 
• The exception/cancellation model is still TBB's. 
• wait() returns void, not task_group_status, since the blog does not detail task_group_status. 

But nonetheless, I think some TBB users will find this minimal form useful.  For example, it's enough of task_group to write the quicksort in Molloy's post.

The code for header follows my signature.  I'd be interested to hear how useful it is.

- Arch

#ifndef __TBB_task_group_H
#define __TBB_task_group_H

#include "tbb/task.h"

namespace tbb {

class task_group;

namespace internal {

// Suppress gratuitous warnings from icc 11.0 when lambda expressions are used in instances of function_task.
#pragma warning(disable: 588)

template<typename Function>
class function_task: public task {
    Function my_func;
    /*override*/ task* execute() {
        my_func();
        return NULL;
    }
public:
    function_task( Function& f ) : my_func(f) {}
};

} // namespace internal

class task_group: internal::no_copy {
private:
    empty_task* root;
public:
    task_group() {
        root = new(task::allocate_root()) empty_task;
        root->set_ref_count(1);
    }
    ~task_group() {
        if( root->ref_count() )
            root->wait_for_all();
        root->destroy(*root);
    }
    template<typename Function>
    void run( Function f ) {
        task& self = task::self();
        self.spawn(*new( self.allocate_additional_child_of( *root )) internal::function_task<Function>(f) );
    }
    void wait() {
        root->wait_for_all();
    }
};

} // namespace tbb

#endif /* __TBB_task_group_H */

TBB tasks rely on non-preemptive cooperative scheduling based on work stealing, similar to Cilk. Once the TBB scheduler starts a task on a software thread, it does not switch to another task except at well-defined points (specifically, while waiting for its child tasks to complete). For compute-bound workloads, this style of scheduling has multiple benefits:

• Cooperative scheduling has low context switch overhead.
• Work stealing has good cache locality and certain space guarantees (see Cilk papers)
• Lastly, and most important, actual parallelism can be matched to available parallelism. In task-based programming, the programmer is expected to provide too much parallelism (“parallel slack” in Cilk parlance), and the scheduler will extract just enough parallelism to keep the machine humming, not swamped.

But programs are not only about calculation.  Programs also often have to wait on external events.  For sake of timely response, the wait needs to be done by a preemptively scheduled thread that can be scheduled when the event occurs. Of course interrupts or polling sometimes work. Interrupts are a bit of a combination of cooperative and preemptive scheduler.  The cooperative part is using an existing thread, the preemptive part is using it any time. That can be the best of both (preemptive and low overhead) or the worse of both (interrupt handlers are typically constrained on what they are allowed to do).  Polling usually scales poorly – composing two polling components requires composing their polling loops or using separate threads for the two polling loops.

Tasks can block, but doing so has two problems:

• The underlying thread sits idle until the task unblocks.
• Tasks (and their threads) waiting on it to also sit idle.  I.e., blocking propagates along a dependence chain.

In an ideal world, the scheduler would fire up other threads to run tasks in the meantime.  To do this efficiently requires user-level scheduling support that is not (at least yet) available in all operating systems targeted by TBB. But even with that support, there is another issue.  K blocked threads consume K stacks. Most of these stacks might be quite small, but current calling conventions require that programmers specify a fixed stack size (or use a default one) that is typically much larger than necessary for the common case. Changing the calling convention to be more like Cilk’s would solve this problem, but calling conventions take a long time to reform.

So in addition to tasks, TBB 2.1 has a class tbb_thread, which is a thin wrapper around a platform’s native thread. The interface is as close to the C++ 200x std::thread as we could make it given the limitations of C++ 1998. In particular:

• Lack of variadic templates restricts us to a fixed limit on template arguments.
• Lack or rvalue references implies slightly more overhead because copy-construction has to be used instead of move construction.
• Time is measured in the existing TBB timing interface tick_count::interval_t instead of the templated time interface in C++ 200x. We had to draw the line somewhere on where to stop pulling in C++ 200x.

We chose to call it tbb::tbb_thread and not tbb::thread to avoid name collisions when the ISO std::thread becomes available and a program liberally employs “using” directives.

Because tbb::tbb_thread is a thin wrapper around native threads, threads are heavier than tasks. They take longer to create and destroy.  They have associated stacks.  They are preemptively scheduled, so they guarantee concurrency, which is useful when you need it, but comes at the price of oversubscription if misused.  But they can block without impacting other threads or tasks. 

So TBB 2.1 has two ways to get things done: tasks and threads. When designing a program, try to separate calculating work from waiting work. Use tasks for calculation and threads for waiting. When a thread needs to do calculations, it can do it with tasks. Avoid having a task block on an external event. Software components doing waiting should call on components doing calculation, not the other way around.

Threading packages often have some notion of a thread id or thread local storage. The two are equivalent in the sense if given one, you can easily build the other. For example, thread local storage can be implemented as a one-to-one map from thread ids to pieces of storage. And vice versa, the address of a variable in thread-local storage can be used as a thread id.

TBB by design has no thread id or thread-local storage. TBB is based on task-based parallelism, where the programmer breaks work up into tasks, and the task scheduler is free to map tasks to hardware threads. Furthermore, our OpenMP run-time group strongly recommended that we avoid explicit thread ids because of problems with nested parallelism and dealing with a dynamically growing or shrinking team of threads. For example, the nature of the OpenMP thread id interface implies that the number of threads in a thread team is fixed for the duration of the team.

However, thread local storage does have its uses. Don Webber posted an excellent use case for thread local storage, which involves updating a sparse matrix. The problem involves doing many updates of the form *p += value, in parallel, where some updates might update the same location. Assuming that += is commutative and associative, one way to implement this is to have each thread sum its own updates privately, and then merge the sums. As Don notes, the alternative of locking *p on each update is prohibitively expensive. Using an atomic +=, even if available on current hardware, would likewise be prohibitively expensive, because cache lines would ping-pong severely.

I'd like to see TBB extended to provide the power of thread local storage without opening up a Pandora's box of raw thread ids. I think the solution needs to cleanly separate a high-level portion from a low-level portion, like we recently did for cache affinity. (Note: the type task::affinity_id might appear to have opened the box, but did not, because it is a hint, not a commandment.)

TBB's template parallel_reduce in TBB partially deals with the cited use case, because it is lazy about fork/joins. The user defines how to fork/join state information for the reduction. The template recursively decomposes the range and applies the users fork/join operations. The laziness is that fork/join pairs are only used when task stealing occurs. For example, if there are P threads and N leaf subranges, it does not do the obvious N-1 fork operations, but instead does just enough to keep the threads busy. Specifically, it does a fork/join pair only when the two children would be processed by different threads.

However, parallel_reduce is not lazy enough. At the high level, the problem is that parallel_reduce cannot assume that the reduction operation is commutative. For a non-commutative reduction operation, the current implementation is close to optimal (maybe a factor of 2 off in the worse case) with respect to the number of fork/join pairs. If TBB added a reduction template that could assume a commutative reduction operation (e.g.parallel_unordered_reduce), then at most P-1 fork/join pairs would be necessary.

The good thing about using the hypothetical parallel_unordered_reduce instead of exposing thread local storage is that it keeps the abstraction at a high level. Explicitly using thread local storage would introduce irrelevant low-level details. For example, a typical implementation based on thread local storage can be sketched as:

forall updates (p,value)  

    do  *p += value // *p points to thread-local partial sum  

for each thread-local partial sum do  

    update global-sum+= thread-local partial sum

This level exposes issues such as "where are the thread-local partial sums that were generated in the first loop?" Since threads can come and go during execution of the first loop, iterating across ids of currently running threads is not enough. Some of the partial sums might outlive their threads, or some threads might come into existence after the partial sums were generated.  We'll need a container to hold the partial sums, and a means of iterating over the sums.

The interface for such a container seems straightforward. Define it as a sequence of T that has iterator capability. Add TBB-style ranges too, so that reductions over the container can be done in parallel. Add a special method "mine()" that returns reference to  the element that is owned by the invoking thread. If the element is not present, "mine()" would insert one and default-construct it. 

Method mine() would be most likely implemented by hashing a thread id, so it's not going to be cheap, but probably inexpensive enough if the user hoists calls to it. 

There is an interesting alternative that weakens guarantees, with the intent of expressing intent at a little higher level. It's somewhat an object-oriented extension of a semaphore that combines the semaphore with the resource it is protecting. It would work as follows. The method "mine() could be replaced by two methods "acquire()" and "release()" [possibly sugared with RAII] such that: 

• "acquire()" would grant access to an instance of T that is not being accessed by any other thread
• "release()" would release access

This interface permits an implementation to keep the limit the number of "thread local" copies of T to what is actually necessary for concurrency, not what is necessary for one-per-thread. If T is really big, this could be advantageous. There's perhaps an issue with cache affinity.  However, a sufficiently clever implementation could bias towards grabbing the instance of T that the thread most recently had before. Of course, a conforming implementation could just use thread-local storage for each copy of T.

Comments?

• We could figure out semantics that are useful for parallel programming and
• We could implement it reasonably efficiently on current hardware.

I usually try to avoid linked lists even for sequential programming if I'm programming for performance. My reasons are:

• Linked lists are often misused in ways that create asymptotic slowness. Usually the culprit is searching the list. Yes, similar abuses can occur for other data structures too. But lists seem to attract this abuse.
• Linked lists are unfriendly to cache. Adjacent items in a list tend to be scattered in memory. With cache misses costing on the order of 100x a cache hit, this can be a significant performance issue.

For parallel programming, add another flaw:

• Traversing a linked list is inherently serial. [Theorists will point out that traversal can be done in parallel if you have a processor per node in the list. Feel free to buy that many Intel processors -- I own Intel stock.]

Two traditional attractions of linked lists are:

1. Linked lists are about the easiest dynamic data structure to write from scratch.
2. Prepending and appending take O(1) time.

But modern polymorphic language like C++ provide dynamic data structures like std::vector and std::deque. You don't have to write them from scratch. Prepending or appending to a deque also takes O(1) time. Appending to a vector takes O(1) amortized time. Amortized time is the time averaged over many append operations.

Here's a speed test you might want to try. Construct a container, append n items, walk the container once, and destroy it. Here's the code:

template<typename Container>

int Iota( int n ) {

    Container container;

    for( int i=0; i<n; ++i )

        container.push_back(i);

    int sum = 0;

    for( typename Container::const_iterator j=container.begin(); j!=container.end(); ++j )

        sum += *j;

    return sum;

}

I tried this fragment on a Linux box and found that std::deque was slightly faster than std::list when n>=3, and std::vector was slightly faster when n>=10. When n>=100, std::deque was more than 10x faster than std::list, and even std::vector more than 3x faster than std::list. So for very short collections, std::list might pay off. But for big collections, its second-rate.

Of course linked lists do have some virtues, notably when concatentating lists, splicing lists, and inserting items in the middle. I use lists when I need to do that. But getting back to parallelism, which set of those operations make any sense in parallel programming? Concurrent splicing and inserting seems awfully tricky to use correctly. For example, if I really need to insert in the middle of the list, it must be because there is something special about the insertion context. But if there are other threads inserting at the same time, how do I know the context will not be broken?

The two operations on lists that I think could be useful in parallel programming are:

1. concatenating two lists, in constant time
2. splitting a list into two sublists, or at least view it as two sublists, in constant time

For example, a parallel reduction could use "concatenate" as its reduction operation, and thus build a list of N items in O(N/P+log(P)) time. The log(P) term arises from a tree reduction at the end. The problem is the second operation. To keep a list from becoming a serial bottleneck, we need a way to traverse it in parallel. That probably means it is no longer a linked list, but some kind of (balanced?) tree structure.

I've had a recurring thought that we should add this kind list, one that supports concatenation and splitting in constant time. But we really need motivating use cases before implementing it. Suggestions for good use cases or demos appreciated.

- Arch Robison

P.S. I though about writing this blog as a politcal attack ad, but given the technical details, decided against it. If I had done it that way, it would have started:

Mr. Linked List is running for office. He's popular everywhere. But here's what Mr. Linked List doesn't want you to know... .