• Intuitive interface: Quickly find issues, without a lot of clutter; GPA’s easy work flow fits the way game developers want to optimize their games.
• In-depth, real-time analysis: Identify bottlenecks, experiment with changes, and see results in real time — all within GPA and without modifying the game code.
• Remote network model: Eliminate the processing overhead of running tools on the same system as your game, thereby improving the overall accuracy of your results (so that you don't end up trying to optimize portions of your code that aren't an issue).
• Extensive API: Extend the tools for your specific needs by adding your own metrics and/or using metrics gathered by GPA in your own analysis tools.
• Intel Integrated Graphics support: Optimize games and graphics-intensive applications for Intel Integrated Graphics–based systems; GPA is the only toolset that can help you optimize your game on these devices.

Scheduling Tasks

Tasks and Task Pool Overview

{
    CTaskCompletion Flag;
    CMyTask* pTask;
	
    pTask = new CMyTask(&Flag,…);
    pThread->PushTask(pTask);
    …
    pThread->WorkUntilDone(&Flag);
}

{
     CMyTask* pTask;
	
     pTask = new CMyTask(pThread->m_pCurrentCompletion,…);
     pThread->PushTask(pTask);
}

Inside the scheduler

A Parallel Game Loop

Bibliography

About the Author

Introduction

Code Characterization, Parallelization

• The concurrency level did not change from iteration to iteration, implying that a very small subset could be chosen for detailed analysis
• The iterations suffer from ending with a critical section, where changes to the game scene get distributed among worker threads before they start working on the next frame

• SMOKE uses an associative container (a map) to collect the change notifications made to game world objects. Find-and-Erase functionality in a thread-safe map introduces noticeable overhead. This overhead can be avoided if the map is replaced by a vector for which each thread has its own range of elements holding notifications for only it to process.
• The number of change notifications is not known in advance, so in order to be able to divide up the vector between all threads, each thread first needs to generate the notifications in Thread Local Storage (TLS), count them, grow the vector atomically to the necessary length, and then concurrently copy change notifications from TLS to the common container. For all of this no synchronization is required.
• After the above changes were implemented it turned out those functions that process different types of change notifications are not thread-safe either. Guarding access to certain common resources fixed this problem.

Results

Summary and Conclusions

Acknowledgments

References

1. SMOKE Game-Technology Demo, Intel Software Network, available at http://software.intel.com/en-us/articles/smoke-game-technology-demo/
2. James Reinders, Intel Threading Building Blocks, O'Reilly Media, Inc, Sebastopol, CA, 2007.
3. Robert D. Blumofe and Charles E. Leiserson, "Scheduling Multithreaded Computations by Work-Stealing," in Proceedings of the 35th Annual IEEE Conference on Foundations of Computer Science, Sante Fe, New Mexico, November 20-22, 1994.
4. Robert D. Blumofe, Christopher F. Joerg, Bradley C. Kuszmaul, Charles E. Leiserson, Keith H. Randall and Yuli Zhou, "Cilk: An Efficient Multithreaded Runtime System," in Proceedings of the Fifth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP '95), Santa Barbara, California, July 19-21, 1995.
5. Michael Voss, "Demystify Scalable Parallelism with Intel Threading Building Block's Generic Parallel Algorithms," DevX.com, Jupiter Media, October 2006, at http://www.devx.com/cplus/Article/32935.
6. Michael Voss, "Enable Safe, Scalable Parallelism with Intel Threading Building Block's Concurrent Containers," DevX.com, Jupiter Media, December 2006, at http://www.devx.com/cplus/Article/33334.
7. Bradley Werth, "Optimizing Game Architectures with TBB", at http://www.gamasutra.com/view/feature/3970/sponsored_feature_optimizing_game_.php.
8. The system used for testing was an Intel X5355: 2x4, 2.66 GHz, 8G RAM, Windows XP x64 Pro SP2, GeForce 8800 GTX. For more info see: http://en.wikipedia.org/wiki/Xeon#5300-series_.22Clovertown.22

About the Authors

*Other names and brands may be claimed as the property of others.

DMIP in IPP 6.1

• Basic arithmetic unary and two-place operations (+, -, *, /, Abs, Ln, Min, Max, Sqrt etc)
• Logical operations (&, |, ~, etc)
• Thresholding operations
• Image type and channels conversion
• Color conversion
• Statistical operations
• Image filters:
  • General filters
  • Special filters (Box, Min, Max, Median, etc)
  • Fixed kernel filters (Sobel, Prewitt, Schar, etc)
  • Morphology (erosion, dilation)
• Linear transform DFT/FFT
• Polyadic operations with up to 5 arguments

IppiSize roi; // Input and output image sizes
Ipp8u* pSrcImg; int srcImgStep; // Input C3, 8u image in IPP image format
Ipp8u* pDstImg; int dstImgStep; // Output C1, 8u image in IPP image format

Ipp8u* pGrayData; int grayDataStep; // Input grayscale image

Ipp16s* pIntBuffer; int IntBufferStep; // Intermediate buffer

Ipp8u* pSBuffer; int SBufferSize; // Sobel operators temporary memory

// Allocate memory for grayscale converted image
pGrayData = ippiMalloc_8u_C1(roi.width,roi.height,&NewSrcStep);

// Calculate and allocate temporary filters memory
ippiFilterSobelHorizGetBufferSize_8u16s_C1R(roi,ippMskSize3x3,&SBufferSize);
ippiFilterSobelVertGetBufferSize_8u16s_C1R(roi,ippMskSize3x3,&SBufferSize1);

SBufferSize = SBufferSize < SBufferSize1 ? SBufferSize1 : SBufferSize;

pSBuffer = ippsMalloc_8u(SBufferSize);

// Allocate memory for intermediate buffer. 
// Its size is twice the image size and is used for dx and dy storage.
pIntBuffer = ippiMalloc_16s_C1(roi.width,roi.height*2,&IntBufferStep);

// Convert input image into grayscaled image
ippiRGBToGray_8u_C3C1R((const Ipp8u*)pSrcImg, srcImgStep, pGrayData, grayDataStep, roi);

// Calculate dx
ippiFilterSobelHorizBorder_8u16s_C1R((const Ipp8u*) pGrayData, grayDataStep,
               pIntBuffer, IntBufferStep, roi, ippMskSize3x3, ippBorderRepl, 0, pSBuffer);
// Calculate dy
ippiFilterSobelVertBorder_8u16s_C1R((const Ipp8u*) pGrayData, grayDataStep,
               pIntBuffer+roi.height*(IntBufferStep/2), IntBufferStep, roi, ippMskSize3x3, ippBorderRepl, 0, pSBuffer);

// Take absolute values of dx and dy.
ippiAbs_16s_C1IR(pIntBuffer, IntBufferStep, roi);
ippiAbs_16s_C1IR(pIntBuffer+roi.height*(IntBufferStep/2), IntBufferStep, roi);

//Add them. Edges are detected.
ippiAdd_16s_C1IRSfs((const Ipp16s*) pIntBuffer+roi.height*(IntBufferStep/2), IntBufferStep, pIntBuffer, IntBufferStep, roi, 0);

// Covert image with edges in dst image format
ippiConvert_16s8u_C1R( pIntBuffer, IntBufferStep, (Ipp8u*)pDstImg, dstImgStep, roi );

// Free resources
ippiFree(pGrayData);
ippiFree(pIntBuffer);
ippsFree(pSBuffer);

Image Src(pSrcImg, Ipp8u, IppC3…); // Source image in DMIP format
Image Dst(pDstImg, Ipp8u, IppC1…); // Destination image in DMIP format

Kernel Kh(idmFilterSobelHoriz,ippMskSize3x3,ipp8u,ipp16s); // Dx operator
Kernel Kv(idmFilterSobelVert,ippMskSize3x3,ipp8u,ipp16s); // Dy operator

Graph O = ToGray(Src); // To get detected as common expression.

// Compile end execute task
Dst = To8u(Abs(O*Kh)+ Abs(O*Kv));

Image Src(pSrcImg, Ipp8u, IppC3…); // Source image in DMIP format
Image Dst(pDstImg, Ipp8u, IppC1…); // Destination image in DMIP format

Kernel Kh(idmFilterSobelHoriz,ippMskSize3x3,ipp8u,ipp16s); // Dx operator
Kernel Kv(idmFilterSobelVert,ippMskSize3x3,ipp8u,ipp16s); // Dy operator

Graph O = ToGray(Src); // To get detected as common expression.

// Build task graph
Graph G = (To8u(Abs(O*Kh)+ Abs(O*Kv))) >> Dst;

// Compile
G.Compile(slice size);
// Execute task
G.Execute(threads number);

Graph optimization technique

// Set desired desired opimization mode
idmStatus Control::SetGraphOptLevel(int level);
// Retrieve current graph optimization mode
int Control::GetGraphOptLevel(void);

Image SrcI(pSrcImg, Ipp8u, IppC1…); // Source image in DMIP format
Image DstI(pDstImg, Ipp8u, IppC1…); // Destination image in DMIP format

Kernel Kh(idmFilterSobelHoriz,ippMskSize3x3,ipp8u,ipp16s); // Dx operator
Kernel Kv(idmFilterSobelVert,ippMskSize3x3,ipp8u,ipp16s); // Dy operator

SrcI.CopyBorder(…);// Create image border.
Graph O = Src(SrcI); // To get detected as common expression.

// Compile end execute task
DstI = To8u(Abs(O*Kh)+ Abs(O*Kv));

DMIP Task examples

Image A(…); // Source image in DMIP format
Image D(…); // Destination image in DMIP format

Kernel K(idmFilterBox,…); // Box filter kernel
Ipp32f c;
Ipp8u Tmax, Tmin;

Graph O = To32f(A); // To get detected as a common expression.

// Compile end execute task
D = Max(Min(To8u(O-(O-O*k)*c),Tmax),Tmin);

Separable 2D FFT

Architecture topology utilization

Many-Core Architecture Challenges

Conclusion

Online Resources

• Intel® IPP Web site:
  (http://www.intel.com/software/products/ipp)
• Intel® IPP Technical Support Resource
  (http://www.intel.com/software/products/support/ipp)
• Intel® IPP Forum:
  (http://softwarecommunity.intel.com/isn/Community/en-US/forums/1274/ShowForum.aspx)
• DMIP Manual (dmipman.pdf) is included with the Intel® IPP product files.
• Dr. Dobb's Journal, 26 Apr 2009. Intel's Integrated Performance Primitives and Deferred Mode Image. (http://www.ddj.com/217100379?cid=RSSfeed_DDJ_All)

About the Authors

Igor Belyakov is a software engineer in Intel's Integrated Performance Primitives Engineering team within the Software and Services Group (SSG). Igor is a technical leader of the DMIP project specializing in automatic optimization and parallelization of image processing and computer vision algorithms on multi-core and manycore architectures. Igor also worked in the Speech Coding Team and is an expert in speech coding and in the real-time media data transmission protocol stack. He holds a MS in Mathematics from the Moscow State University, Mathematic and Mechanical Department.

Bonnie Aona is a software engineer in the Intel Compilers and Languages Group within the Software and Services Group (SSG) working on optimizing and testing applications to take advantage of the latest Intel software and hardware innovations to achieve high performance and parallelism.  Bonnie's career leverages Software Quality Assurance and program management with software design for complex high performance applications for computer graphics, real-time systems, scientific research, manufacturing, e-Commerce, aerospace and healthcare.  She holds Masters degrees in Electrical and Computer Engineering from University of California at Davis.

While working with an early Cilk++ adopter, it quickly became apparent that the default memory allocator shipped with the Windows C Run-Time Library can be a bottleneck in a multithreaded application. The Windows memory allocator has a single lock which it uses to serialize access to its internal structures. While this is safe, it proved to cause a serious loss of parallelism in the customer’s application.

There are lots of memory allocators available, both open source and commercially. We looked at using a number of them didn't find any that had the combination of features we were looking for. Ultimately we chose to write our own. Since the customer's program was a Windows application, we initially wrote concentrated on the Windows implementation.

Features borrowed from Hoard

We decided to build the new allocator inspired by the principles in Hoard: A Scalable Memory Allocator for Multithreaded Applications by Emery D. Berger, Kathryn S. McKinley, Robert D. Blumofe and Paul R. Wilson. Hoard is designed to minimize lock contention between threads, false sharing and memory drift. Detailed information about Hoard can be found at the Hoard website.

We named our new memory allocator "Miser" to play on its relationship to Hoard. We have implemented Miser for both Linux and Windows, and this post describes data structures common to both, several Windows-specific implementation details, and several Linux-specific notes at the end.

Superblocks

The basic unit of allocation in Miser is a "superblock." A superblock holds a header, an array to hold the size of each requested block of memory and an array of same-sized memory blocks, or bins, which can be used to satisfy memory requests. The bins are powers of 2; 8 bytes through 256 bytes. Studies have shown that this will satisfy over 98% of memory requests. Requests for larger blocks are forwarded to the C Run-Time Library. The array of bins is placed against the back of the superblock, so memory blocks returned to the user are always naturally aligned.

Global and Per-Thread Pools

Miser, like Hoard, features a private pool of memory for each thread. Because the pool is thread-private, it can be accessed without locking. A pool is made up of superblocks which are kept in a roughly sorted order based on how much space is available in each superblock. The use of the per-thread pools is fundamental to improving application performance:

• Per-thread pools allow Miser to satisfy most memory requests without locking.
• Per-thread pools limit false-sharing. Unless a block is passed from one thread to another, all use of the memory accessed through a cache-line should be from one thread.

The global pool is not owned by any thread. If a per-thread pool cannot satisfy a request, it will take a superblock from the global pool. If the global pool is empty, Miser will allocate more memory from the operating system. As memory is released to a per-thread pool, it will contribute superblocks to the global pool to prevent memory drift.

Additional Features

While the features inspired by Hoard provided a good foundation, we had the following additional requirements for Miser:

• It must be able to be loaded dynamically. This allows a component shipped as a shared library to use Miser instead of requiring that the application load it at startup.
• It must be able to recognize blocks that it had not allocated, and pass those to the system C Run-Time Library.
• It must be thread-safe and fast.
• It must be able to simultaneously support the multiple versions of the C Run-Time Library that may be in use by an application.

Loading Miser dynamically

The format for a Windows module, either an executable or Dynamic-Link Library (DLL) is defined by the Windows Portable Executable File Format. Almost every Windows module imports functionality from some other module. These dependencies are described by a pair of tables; the Import Name Table and the Import Address Table. When a module is loaded into memory, the loader will "snap" the addresses in the Import Address Table to the entrypoint in the destination module. All calls to that function from the module will be made indirectly through this cell.

When Miser is loaded into an application, it will go through each of the modules in the application and redirect the entries in the Import Address Table for each of the C Run-Time Library memory allocation functions to its own entrypoint. This technique was first described by Matt Pietrek in Peering Inside the PE: A Tour of the Win32 Portable Executable File Format. Since the modification of the IAT entry is a 32-bit write, it’s an atomic operation; the IAT entry is either the C Run-Time Library address or the Miser address.

Once Miser has been loaded into a process, it can never be unloaded. While Miser could restore the original values of the C Run-Time Library functions in the IATs, the C Run-Time Library functions cannot handle memory allocated by Miser.

Recognizing blocks not allocated by Miser

When Miser loads into the application, the application may have already allocated memory. This is no way for Miser to replace those blocks with blocks from its own resources. Since Miser redirects all calls to free() and _msize() to itself, it must be able to recognize memory that has been allocated by the C Run-Time Library and pass those calls to the appropriate version of the C Run-Time Library.

The Windows OS provides a number of user-mode APIs to manage memory. One of the most fundamental is VirtualAlloc() which allocates memory in 64K blocks on 64K boundaries. Miser will subdivide these into 4K superblocks, each of which starts on a 4K boundary. Not coincidentally, this is the page size for the 32-bit x86 Windows OS. The first thing in each superblock is a header which contains a "magic number" and other control information. So by masking off the low 12 bits of a memory address, we should find the header for the superblock containing the memory block. There are two checks to validate a superblock.

1. The superblock header contains the correct "magic number."
2. The superblock header contains a pointer to the per-thread pool owning the superblock and an index into an array of pointers to the superblocks owned by the per-thread pool. The entry in the array should match the address of the superblock.

If either of these tests fails, Miser will pass the call off to the Windows C Run-Time Library.

An additional benefit of being able to detect memory allocated by the C Run-Time Library and pass the request on is that Miser can pass any requests for more than 256 bytes off to the C Run-Time Library.

Supporting multiple versions of the C Run-Time Library

Each module loaded into an application can be built against a different version of the C Run-Time Library. In addition to the side-by-side support built into the OS, recent versions of the C Run-Time Library have had their version number in the DLL name. Miser supports all versions of the C Run-Time Library since Visual Studio .NET 2002. Since each version of the C Run-Time Library has its own heap, Miser keeps track of which version of the C Run-Time Library it has hooked and will forward calls to the appropriate version.

Speed and thread-safety

Thread-safety and speed are intimately tied. The more the code needs to take out a lock, the greater the chance that it will have to wait for some other thread. For most operations, Miser does not need to lock; it can usually satisfy a request from resources already available in the per-thread pool. Miser must take out a lock in the following circumstances:

1. Accessing the global pool – only one thread may be allowed to access anything in the global pool at any time. Any thread that attempts to access the global pool must lock it first, serializing access.
2. Freeing a block allocated from another per-thread pool – Unlike Hoard, Miser does not maintain a lock in the superblock header. Blocks to be freed in some other per-thread pool are placed on that per-thread pool's remote free queue using interlocked instructions. The remote free queue is drained before attempting to allocate a block in the hopes that a freed block may satisfy the request.
3. Supporting _msize() for a block allocated by another thread - The _msize() function allows a Windows application to query the C Run-Time Library about the size of a block of memory. The value returned is the size in bytes that was specified when the block was allocated. If the block was allocated by Miser in the current thread, we can safely access the information in a superblock owned by the per-thread pool. However if the superblock is owned by another thread’s per-thread pool, then we must lock the per-thread pool before we attempt to validate the superblock against its superblock array.
4. Maintaining the list of superblocks – Each per-thread pool maintains an array of the superblocks that it has allocated. The header for each superblock has a pointer to the per-thread pool that owns it, as well as an index into the per-thread pool superblock array. These are used to validate that a memory address is from a block that Miser allocated. Because another thread may access the array of superblocks to satisfy an _msize() call, the array of superblocks must be locked before it is updated.

One important thing to note is that while Miser supports Cilk++ well, it is not tied in any way to the Cilk++ compiler or runtime. It should serve equally well for a multithreaded application using any of the threading technologies available.

Miser Limitations on Windows

• Miser does not support any of the C Run-Time Library calls which allocate memory on an address boundary.
• Miser assumes program correctness. It does not put any padding between blocks to attempt to detect writes beyond the bounds of the allocated block. Allocated blocks are butted directly against each other; Miser keeps the allocation size in a separate structure to allow it to maintain natural alignment in it's bin arrays.
• Because of its dependence on the Import Address Table to hook into a module, Miser can only be used with modules that use the DLL forms of the C Run-Time Library. Modules that link against the static C Run-Time Library do not go through the IAT, so Miser cannot hook into them.

Performance

The following is an example of performance gains achievable with Miser, from an earlier blog post (Multicore-enabling the N-Queens Problem Using Cilk++).

Miser on Linux

Although Miser was initially written for Windows, since it performed better than Hoard in some of our tests, it was ported to Linux. The back-end code remained basically identical except for various system calls (e.g., changing calls to VirtualAlloc() to mmap(), etc.).

On the front-end, the mechanism for accessing the functions was rewritten. The Miser library exports malloc(), free(), calloc(), etc., and linking a binary with it will cause the loader to look for it at runtime. Additionally, preloading it by setting
LD_LIBRARY_PATH causes objects to find Miser's allocators before those of the C runtime library. Miser still uses the system
malloc() for large requests and it finds them in the default library using dlsym().

In Linux, if Miser is loaded after a module has resolved the location of allocator functions, Miser does not hook itself in. You can still access Miser's malloc() using dlsym() and getting "malloc" from the library but, as in Windows, memory allocated with Miser must be freed with Miser. The system free() doesn't know what to do with Miser memory.

1. thread safety,
2. overhead,
3. contention,
4. memory drift.

Thread safety

Basic storage allocators are not thread safe, although recent efforts have started to remedy this problem for many concurrency platforms.  In other words, improper behavior due to races on the storage allocator's internal data structures can result from two parallel threads attempting allocate or deallocate at the same time.  When threads have unrestricted access to the storage allocator, as shown below, they may end up "stomping on each others' toes," leading to anomalous behavior.

The simple solution to this problem is for applications to acquire a mutex (mutual exclusion) lock on the allocator before calling malloc() or free(), as illustrated below, which lets only one thread access the allocator's internal data structures at a time.

If the storage allocator is thread safe, the locking protocol is incorporated into the logic of the storage allocator itself.

Overhead and contention

Two problems may arise when an allocator is made thread safe by locking.  The first is that allocation and deallocation may now be slower due to the overhead of locking.  The second is that contention may arise in accessing the storage allocator, which can slow down the application and limit its scalability.  Contention may not be a big problem for 2 or 4 cores, but as Moore's Law brings us dozens and even hundreds of cores per chip, contention can threaten scalability.

Both problems can be solved using a distributed allocator, which provides a local storage pool per thread, as illustrated below.

A distributed allocator allows allocation and deallocation to run out of the local storage pool most of the time.  In the uncommon case that a thread's local pool is exhausted, the thread can obtain additional storage, typically in large blocks, from the global pool.  The contention problem is solved, because threads only rarely access the global pool.  The overhead problem is solved as well, because no locking is needed to access the local pool.

Memory drift

Unfortunately, local pools introduce yet another problem, especially in concurrency platforms where storage is actively shared among threads or which load-balance a computation across the threads.  One thread A may continually allocate storage out of its local pool and pass it off to another thread B which frees it into its local pool.  When thread A's local pool runs out, it allocates more storage from the global pool.  This storage is passed to B, which proceeds to free it into its local pool.  Over time, B's local pool grows unboundedly, creating something akin to a memory leak, where the virtual-memory footprint of the application continues to grow.

This memory drift problem can be solved in two ways.  One solution is for a thread whose local pool becomes too large to return some of its storage to the global pool.   The other is for all threads to return storage to the thread pool where the storage was allocated.  Either method can be implemented with low overhead, and both provide satisfactory solutions to the memory drift problem.

Conclusion

There are other problems that can arise with parallel storage allocators.  For example, false sharing is a particularly pernicious problem, where two threads access independent blocks of storage that happen to lie on the same cache line, leading to a thrashing of the cache coherency protocol in the processor.  A storage allocator that fails to respect cache line boundaries and gives blocks of storage that share the same cache line to different threads may induce false sharing, which is hard to detect, because the logic of the code shows that the threads are accessing independent locations.

Two examples of parallel storage allocators include Hoard, written by Emery Berger of the University of Massachusetts, and the Miser allocator, distributed by Cilk Arts as part of our Cilk++ distribution.  (More on Miser in an upcoming post - stay tuned!)

This simple but powerful principle guides more than just the appearance of Cilk++ programs. It also simplifies the process of developing and testing a Cilk++ program, and it allows Cilk Arts to provide powerful, efficient, and provably correct tools.

(Serial semantics should not be confused with sequential consistency, which is a concept invented by Leslie Lamport and discussed extensively in the literature of parallel computing. A parallel computing system with sequentially consistent memory guarantees that, in Lamport's words, "... the result of any execution is the same as if the operations of all the processors were executed in some sequential order, and the operations of each individual processor appear in this sequence in the order specified by its program." In contrast, serial semantics means that the program can be executed using a single thread of control as an ordinary serial program - one does not need a conceptual parallel model to understand -- or execute -- the program.)

Cilk++ Serial Semantics

First, let's take a closer look at how Cilk++ provides serial semantics.

Cilk++ provides three new keywords: cilk_spawn, cilk_for, and cilk_sync. Each of these has an intuitive serial interpretation:

As you can see, a Cilk++ program can be converted to a serial program quite easily: simply replace cilk_for with for, and delete the cilk_spawn and cilk_sync keywords. The Cilk++ development system provides a compile-time switch to build the "serialization" of a Cilk++ program. Using Cilk++ for Linux, the option [-fcilk-stub] is used with cilk++ to "stub out" Cilk features. Using Cilk++ for Windows, run cilkpp with the [/cilkp cpp] option.

Okay, I hear you saying, "Fine, your fancy parallel program can devolve into a serial program. So what? If I wanted the serial version, I would have just used C++ in the first place!" Glad you raised the issue! A parallel programming model with serial semantics maintains four real advantages over program models without serial semantics.

1. The equivalent serial C++ program can easily be debugged and analyzed using existing development tools. This is an ideal way to debug a parallel program without worrying about parallel interaction, and without needing to consider multiple stacks and program counters. Also, as vanilla C++, all symbol and debug information is available to standard system and third-party tools. Tools that are easy to run on the serialization but might require custom versions to operate on the parallel program include performance profilers, code coverage tools, security analysis tools, memory leak checkers, and any other tools that work with C++ programs.
2. The serial semantics of Cilk++ programs allows Cilk Arts to build better tools that analyze the performance and correctness of Cilk++ programs, and these tools offer strong guarantees about the program. For example, our Cilkscreen race detector runs the test program in a serialized mode on a single processor. With information about the parallel structure of the program (i.e., where the spawns and syncs occur), Cilkscreen can analyze all possible schedules of the program that might execute on any number of processors. In other words, Cilkscreen can detect whether race conditions exist that could manifest under any possible execution of the program. The serial semantics of Cilk++ permits Cilkscreen to analyze the parallel program while running in a single worker (the Cilk++ name for a thread of execution) on a single processor.
3. The serial semantics of Cilk++ provides a real advantage to program testing and quality assurance. Cilk++ is designed to ensure that parallel programs written in Cilk++ can be deterministic, reliably producing results identical to the serialization. This guarantee makes it easy to leverage traditional testing tools that assume that multiple runs of a program always return the same result if given the same input. The serial semantics of Cilk++ eliminates the need to consider timing and scheduling considerations when evaluating program correctness, and it makes it simple to compare the output of multiple program runs for consistency. Repeatability is one of the key properties that make Cilk++ programs testable and reliable.
4. Serial semantics provides high performance while allowing you to design your program without specifying the number of processors on which the program will execute. The Cilk++ runtime scheduler takes care of efficiently load balancing your program across however many processors are available. When a portion of your parallel computation executes on a single processor, Cilk++ can execute it just like ordinary C++ code, taking full advantage of all the compiler optimizations and runtime efficiencies that a good C++ system offers. By starting from good single-core performance, Cilk++ ensures that a program with sufficient parallelism gets good speedup whether it is run on a large number of processors or just a few.

What's the Catch?

Perhaps this all sounds too good to be true - and maybe now you're thinking, "Okay, so what‘s the catch?" All right, there are a few things I've glossed over. For one, since there are some kinds of parallelism that actually don't have serial semantics, you can't write programs using these parallelization strategies in Cilk++. For example, Cilk++ doesn't support producer/consumer, software pipelining, or message passing. Nevertheless, although we're giving something up to have to serial semantics, our experience is that less is more. The four advantages outlined above readily make up for any loss in generality.

There's a second catch, however. Note that in the third point above, I said that Cilk++ programs can be deterministic, not that they are deterministic. How so? Well, the output of a program with a determinacy race (see "Are Determinacy Races Lurking in Your Multicore Application") can depend on how the program is scheduled on multiple processors, and the race condition that leads to this nondeterministic behavior is usually a bug. In most programming environments, you'd be stuck at this point trying to find the race bug by laborious program inspection. In contrast, in the Cilk++ programming environment, Cilkscreen can help you find and eliminate these races, giving you your determinism back. Thus, this "catch," though problematic, can be dealt with effectively precisely because of serial semantics: in effect, Cilkscreen uses the serialization as a benchmark for correctness of a parallel execution.

Sometimes, however, a Cilk++ program is intentionally nondeterministic. For example, some search programs use a concurrent hash table to "remember" the results of intermediate subsearches so that the subsearch doesn't have to be reexecuted if it reoccurs. In this context, each slot of the hash table may contain a mutual-exclusion lock to allow parallel branches of a computation to safely access and update the items stored in the slot. Cilkscreen correctly ensures that races are not declared on operations protected by the same lock.

In practice, however, we have found that most Cilk++ programs require few, if any, locks. In particular, Cilk++'s parallel control constructs obviate the need for the locks that other parallel programming models require for interthread communication and synchronization. Moreover, for many common situations where locking would seem to be necessary to avoid a race, Cilk++ provides hyperobjects, a novel data structure that can resolve races on global variables without sacrificing performance or determinism. By avoiding locks and using hyperobjects, Cilk++ sidesteps many performance anomalies caused by locks, such as lock contention, which can slow down parallel programs significantly, and deadlock, which may cause your application to freeze midexecution.

The Bottom Line

To sum up, Cilk++'s serial semantics provide three key benefits - performance, reliability, and productivity:

• Scalable performance comes from a model that doesn't rely on the programmer knowing the number of available processors in advance. Developers can use traditional serial performance tuning tools to find hotspots and improve both serial and parallel performance.
• Reliability is enhanced through the testing advantages of repeatability and determinism, as well as the ability to build an efficient and provably correct race detector that can compare the parallel semantics of the program with the serial semantics provided by the program's serialization.
• Finally, programmers can be more productive when they use familiar paradigms for new development, and of course, legacy serial code can be converted into Cilk++ easily, because it does not need to be rewritten to adopt a completely different programming model such as data-parallel or message-passing styles.

Tags: serial semantics

Because some applications have as little as 1,000 lines of code while others have 500,000 or more, there is a good deal of variation in the calendar time involved. The project can take as little as a couple days, or stretch to a couple months. But the workflow itself - illustrated below - is fairly consistent from project to project: start with the serial code and verify its correctness (steps 1 and 2); identify and address performance bottlenecks by adding Cilk++ keywords (steps 3 and 4); verify serial and parallel correctness, potentially employing hyperobjects to resolve races (steps 5 through 7); and potentially repeating this process, guided by performance targets.

Because the initial projects typically target existing serial applications, the early Cilk++ projects typically jump to step 3, and the first question is often, "Where are the performance bottlenecks?" And shortly thereafter, "Did exposing the parallelism create a race condition?" Fortunately, there's some great tools available to answer both questions. Let's take a look at a recent example.

Identifying performance bottlenecks

As an example of zeroing in on a hotspot, consider the case of Dan Mirman, a UConn Psychology Postdoctoral Fellow investigating how word meanings are organized and structured in the brain. As part of his research, Dan uses the MikeNet Neural Network Simulator library.

Dan's goal was to accelerate the research feedback loop. Running each simulation takes days, and dozens of neural network simulations are required for each research project. Speeding up simulations from days to hours could have a great impact on the research feedback loop. Software engineers at Persistent Ltd (a Cilk Arts partner interested in providing multicore enablement services to their customers) used Intel's VTune Performance Analyzer software to zero in on the hotspots in the code.

VTune helps you identify and characterize performance issues by collecting performance data from your application, organizing and displaying the data in a variety of interactive views (from system-wide down to source code or processor instruction perspective), identifying potential performance issues and suggesting improvements.

The sampling source view displays source code annotated with performance data. It turned out that three functions were responsible for 97% of the compute time:

Exposing Parallelism with Cilk++

Again with VTune's help, we can quickly drill down to the very lines responsible for the delay.

In this case, it was the outer for loops in the three functions. It was a trivial effort to multithread them by replacing the serial for loops with cilk_for loops:

// parallelized

void mikenet_matrix_vec_mult_t_p (Real * outvec,int nout,Real *invec,

int nin,Real **mat) {

cilk_for (int i = 0; i < nout; ++i) {

for (int j = 0; j < nin; ++j) {

outvec[i] += mat[j][i] * invec[j];

}

}

}

The following illustrates the performance gains achieved by multithreading MikeNet (4-core 1.7GHz system with 2GB of RAM and 512 KB cache/core).

3.5X improvement on 4 cores is pretty darn good...but can we do better? It turns out we can: in one of the functions every memory access misses the cache, and there is thus an opportunity for a more cache-friendly algorithm.

Swapping the inner and outer loop results in a more cache-friendly algorithm:

// inverted, parallelized (with a race on outvec).

void mikenet_matrix_vec_mult_t_p (Real * outvec,int nout,Real *invec,

int nin,Real **mat) {

cilk_for (int j = 0; j < nin; ++j) {

for (int i = 0; i < nout; ++i) {

outvec[i] += mat[j][i] * invec[j];

}

}

}

But when we run the Cilkscreen race detector, we discover that there is now a race on outvec.

Fortunately, Cilk++ reducer hyperobjects are ideally suited for eliminating this race condition without requiring a lock on outvec. Here's the new code:

// inverted, parallelized with reducer.

void mikenet_matrix_vec_mult_t_p (Real * outvec,int nout,Real *invec,

int nin,Real **mat) {

array_reducer_t art(nout, outvec);

cilk::hyperobject<array_reducer_t> rvec(art);

cilk_for (int j = 0; j < nin; ++j) {

Real *array = rvec().array;

for (int i = 0; i < nout; ++i) {

array[i] += mat[j][i] * invec[j];

}

}

}

The cache-friendly algorithm picked up a factor 3X relative to the serial implementation on a single processor, and on 4 cores reached 8X relative to the original serial version (for direct comparison, the data from the previous chart is included in the chart below). As usual, improvement to a serial algorithm paid dividends when multicore-enabling it.

Conclusion

A key part of a multithreading project is figuring out early on where the performance bottlenecks are, exposing parallelism, and then verifying that the parallelism has not created a race condition. The combination of familiar profiling tools such as VTune, and the Cilkscreen race detector, take a lot of guesswork out of multithreading.

Your computer's cache is divided into lines. When your CPU accesses a certain memory address, if it isn't in the cache, it will fetch a line from the next level out rather than a single word. This is a slow process, but if subsequent accesses to memory are nearby, there is a high probability that what the CPU needs is already in the cache. However, certain structures are unlikely to fit in the cache all at once, or at least they may be spread across many lines. Matrices are a prime example and the consequence is that the way in which you access elements of a matrix has a significant bearing on performance. Consider the following two C++ snippets accessing the same matrix along different dimensions:

for (int i = 0; i < rows; ++i) {
    for (int j = 0; j < cols; ++j) {
        out[i] += matrix[i][j]) * in[j];
    }
}

for (int i = 0; i < cols; ++i) {
    for (int j = 0; j < rows; ++j) {
        out[i] += matrix[j][i] * in[j];
    }
}

The two snippets have substantially different performance, which I'll illustrate with a recent example from an early adopter of Cilk++. Dan Mirman, a UConn Psychology Postdoctoral Fellow, brought his neural network simulation code to us because each time he ran it, it would take him a day to get results. Naturally, he wanted to parallelize it to improve performance. The two snippets above correspond directly to two functions in the MikeNet Neural Network Simulator library.

The former function is used to multiply a matrix by a vector, and the latter apparently was copied and pasted and the indices were reversed to perform the multiplication of a matrix transpose by a vector. In fact, there was a trivial modification, taking into account cache usage, that turned out to be nearly as significant as parallelization. To understand the nature of the change, consider the following layout of a matrix in memory:

Sequential accesses in the first function point to adjacent values in memory. Thus, accesses are quick. And the only time it misses the cache (on matrix accesses) is when it is done with the old line and will never load it again.

In the second function, rather than iterating through sequential addresses, every access is on a different line. In short, every access misses the cache. Furthermore, by the time the outer loop completes the first iteration, the second iteration's memory accesses are looking for lines that have long since been flushed. In other words, where the first function pulls each line into the cache once, the second pulls each line into the cache for every iteration of the outer loop.

With this in mind, a simple modification to MikeNet achieved ~3.4x improvement to the overall program on a roughly 500x500 matrix without using Cilk++ at all! All that was required was to invert the two for-loops in the transpose function so that sequential iterations of the inner loop were accessing adjacent locations in memory:

for (int j = 0; j < rows; ++j) {
    for (int i = 0; i < cols; ++i) {
        out[i] += matrix[j][i] * in[j];
    }
}

Incidentally, if you use MikeNet or otherwise have similar matrix code, this is a simple modification that you can perform to save yourself some cycles. This is already enough to change the way Dan interacts with his program. But can the work be spread across multiple cores?

The original algorithm has some hidden parallelism in both functions that is easy to expose. In the original code, merely converting the outer for-loops into cilk_for-loops provides some limited parallelism. The change is not quite as trivial in the second function after it has been modified for cache locality, however. Since the output vector is based on the old inner loop, and since it is the new outer loop, parallelizing it causes a data race on each of the elements in the vector. Furthermore, so little work is done in the new inner loop that the overhead to running it in parallel is significant. This, of course, is precisely the purpose of a reducer. Wrapping the output vector in a reducer eliminates the data race caused by parallelizing the new outer loop:

array_reducer_t art(out, cols);
cilk::hyper_ptr art_ptr(art); 
cilk_for (int j = 0; j < rows; ++j) {
    float *tmp = art_ptr->get_value();
    for (int i = 0; i < cols; ++i) {
        tmp[i] += matrix[j][i] * in[j];
    }
}

This reducer is about as expensive as a reducer can be. Every time it merges, it must walk down two vectors and sum each pair of elements (complexity: O(cols)). In general, one would much rather have reductions that perform trivial operations to cut down on overhead.Nevertheless, when I made the changes to the algorithm, even with the expensive reducer, and with the limited parallelism the cilk_for exposes, I saw significant performance improvement on one of our testing servers.

Bear in mind that I parallelized three functions in the whole program, and the only difference between the two versions is that one of the functions has been altered to access its matrix along a different dimension.

To sum up, identifying parallelism is a key part of program analysis, but it isn't the only thing you should consider when thinking about performance. Especially for some of these large data structures (e.g., matrices), taking some time to look critically at how they interact with your computer's cache is just as significant.