Welcome to 1024cores!

Currently there are some materials on fundamentals of synchronization algorithms, articles on some practical problems (reader-writer problem, producer-consumer queues, lazy concurrent initialization), some materials on scalable architecture, collection of my write-ups for Intel Threading Challenge 2009/2010 and some other not so developed sections. However the site is basically 10-days old and much more is coming, so I encourage you to subscribe to the RSS and/or follow the blog.

Stay tuned and keep threading!

I've just read the article Use Thread-local Storage to Reduce Synchronization from the Intel Guide for Developing Multithreaded Applications and here is my take on it.

Indeed. If threads are a tool for concurrency, synchronization is a tool for suppressing concurrency. Any form of synchronization (no matter locks, semaphores or atomic operations) hinders concurrency. It requires a distributed system to achieve strong global consensus, and consensus in a distributed system can't be cheap. Period.

So the best design of a concurrent system tries to reduce synchronization to a bare minimum (total elimination of synchronization is impossible, otherwise the system will break up into several independent systems). There are various techniques for reducing the need for synchronization - partitioning, privatization, replication, amortization.

The idea of partitioning is to split whole data-set into several mostly independent partitions, a worker thread is bound to each of the partitions, plus there must be some partitioning function that maps an external key to a partition where the data resides. Then, all requests are routed directly to a thread that bound to the required partition. As a result, a worker thread works with the partition's data without any synchronization.

Privatization is a private case of the partitioning with a single partition, i.e. whole data-set is handed over to a single thread which can work with it without any synchronization. The negative side of this technique is that the single thread can become a bottleneck, other threads concurrently running on other cores can overwhelm it with requests.

The idea of replication is to have several independent replicas of a data-set, and propagate updates between replicas explicitly via messages. Data in replicas can be temporary inconsistent, however a lot of systems can tolerate some inconsistency.

Amortization is usually based on some form of thread-local data (placed either on a thread's stack or in a compiler/OS-provided storage). The idea is simple – we collect some updates in thread-local storage and then apply them later in batches. That's what we saw in the article. The main advantage of amortization based on thread-local storage is it's simplicity. Indeed, you do not need to reorganize your data, to route requests to particular threads based on data placement, cope with inconsistencies, etc. So, if it's applicable it's the first thing you must consider.

Well, there are too many things I can say on these things... a way too many to fit into this blog. But what I want to communicate is that you must consider these things as a starting point rather than a final destination, they are a primitive tools for reducing synchronization in your concurrency toolbox. Choose the best tool for a particular situation, combine them, adopt them.

Now a few comments directly on the article.

This solution trades synchronization per event for synchronization per thread. Performance will improve if the number of events is much larger than the number of threads.

I would not agree here, there is no such a tradeoff involved. If a thread had not collected any events in his thread-local storage, then he just does not access centralized data at all. The additional overhead is a single 'if' statement per thread, which is negligible in a context of inter-thread work distribution. This technique does not increase the total number of events.

An additional advantage of using thread-local storage during time-critical portions of the program is that the data may stay live in a processor’s cache longer than shared data, if the processors do not share a data cache. When the same address exists in the data cache of several processors and is written by one of them, it must be invalidated in the caches of all other processors, causing it to be re-fetched from memory when the other processors access it. But thread-local data will never be written by any other processors than the one it is local to and will therefore be more likely to remain in the cache of its processor.

In general this is very true. Indeed, thread-local data reduces amount of inter-core communication, thus reducing amount of costly cache-coherence traffic.

But, this has little to do with shared cashes, even if cores share L3 cache, data still will be transferred between their L1 caches (L1 caches are not shared between cores on most current processors). So I would recommend to just ignore the part on shared caches. Prefer thread-local data and you are on the safe side with any current or future architecture.

There is another important consideration with regard to shared L2/L3 caches (which are featured on many current processors), and this consideration is against thread-local data. Consider the following situation. Moderate size shared object is frequently accessed for reading, but infrequently for writing. If it is split into thread-local parts (which usually implies increase in size), it will not fit into shared L2/L3 cache, thus threads will constantly evict each others data from the cache. However, if the object is implemented as a single centralized entity, it fits into the cache, thus threads will work with cached data without evictions.

So, the tradeoff frequently involved is reduction of synchronization versus increase of total working set. Which to prefer is highly dependent on the situation.

One must be careful about the trade-offs involved in this technique. The technique does not remove the need for synchronization, but only moves the synchronization from a time-critical section of the code to a non-time-critical section of the code.

Well, I would say that the main point of the technique is reduction of synchronization rather than move of the synchronization from one part of the code to another. Amortization via thread-local storage can not involve any movement of synchronization at all. The technique can be applied in two forms: single final aggregation or periodic aggregations. The latter does not involve any movement of synchronization while still reduces synchronization overheads. And it has additional benefit that separate monitoring thread can periodically fetch and output intermediate results.

Consider, for example, the following program:

long total_event_count;

__declspec(thread) long thread_event_count; // thread-local cache

void thread_function(size_t begin, size_t end)

{

 for (size_t i = begin; i != end; i += 1)

 {

  if (predicate(i))

  {

   thread_event_count += 1;

   // if we have cached enough events,

   // transfer them to global shared variable

   if (thread_event_count == THRESHOLD)

   {

    _InterlockedExchangeAdd(&total_event_count, thread_event_count);

    thread_event_count = 0;

   }

  }

 }

  // transfer the remainder of locally cached events

 if (thread_event_count)

 _InterlockedExchangeAdd(& total_event_count, thread_event_count);

}

In the above example the synchronization is not moved to another point, but it's still reduced by a factor of THRESHOLD. Separate monitoring thread can periodically read and output total_event_count variable, and there is a guarantee that total_event_count does not lag behind real value of discovered events by more than NUMBER_OF_THREAD * THRESHOLD.

Note that thread-local data may be actually shared between threads, there is nothing preventing this. A method of declaration of a variable is orthogonal to it's “shared-ness”. Address of a variable declared as __declspec(thread)/__thread/omp threadprivate/pthread_key_create()/TlsAlloc() can be passed to another thread, and thus the variable become shared. Just as plain global variable can be ever accessed by a single thread, and so it's local to the thread.

Also note that you can get a flavor of thread-local data with plain global array indexed by a unique thread index. This technique is less dependent on a particular compiler/OS, and makes sharing of thread-local data much easier (infrequent sharing is not dangerous and anyway necessary in any real-world program). Here is a simple example:

// array of "thread-local" data

long volatile event_counts [MAX_THREAD_COUNT] = {};

// sequence used to generate unique thread indexes

long volatile thread_sequence = 0;

// worker thread routine

void worker_thread(size_t begin, size_t end)

{

 // obtain unique thread index

 long my_idx = _InterlockedIncrement(&thread_sequence) - 1;

 for (size_t i = begin; i != end; i += 1)

 {

  if (predicate(i))

  event_counts[my_idx] += 1;

 } 

}

// monitoring thread routine

void monitor_thread()

{

 while (termination_condition == false)

 {

  // obtain current thread count

  long thread_count = thread_sequence;

  long sum = 0;

  for (long i = 0; i != thread_count; i += 1)

  sum += event_counts[i];

  printf("event count: %u\n", (unsigned)sum);

  Sleep(1000);

 }

}

However, be aware that the above example contains a nasty instance of false-sharing which kills performance. You can read about how to cope with it in the article Avoiding and Identifying False Sharing Among Threads.

Keep threading!

Line Segment Intersection Problem

1. Problem Statement

Write a threaded code to find pairs of input line segments that intersect within three-dimensional space. Line segments are defined by 6 integers representing the two (x,y,z) endpoints.

2. Single-threaded Implementation

2.1. Algorithm outline

As a base algorithm I choose simple exhaustion of segment pairs:

void solve_segment_intersection(std::vector<segment_t> const& segments, std::vector<intersection_t>& results)

{

    for (size_t i = 0; i != segments.size(); i += 1)

    {

        for (size_t j = i + 1; j != segments.size(); j += 1)

        {

            if (is_intersect(segments[i], segments[j]))

                results.push_back(intersection_t(segments[i], segments[j]));

        }

    }

}

Computational complexity of such algorithm is O(N^2 / 2). It's quite high complexity, and there are known algorithms with complexity O((N+K)*logN) [1] and even O(N*logN + K) [2] (K - number of intersections). However I decide to not implement these theoretically more efficient algorithms because of the significantly higher implementation complexity and the fact that they are much harder to parallelize efficiently and less amenable to low-level optimizations such as vectorization, branch elimination, cache-conscious memory access patterns, etc.

In order to reduce computational complexity I apply sorting of segments by X coordinate with following bounding of exhaustion. If segments are sorted by X1 coordinate (it's assumed that X1 <= X2), then once we encounter segment with X1 greater then X2 coordinate of original segment we may not consider all subsequent segments (they are not overlapping with original segment by X coordinate):

void solve_segment_intersection(std::vector<segment_t>& segments, std::vector<intersection_t>& results)
{
    std::sort(segments.begin(), segments.end(), sort_by_x1);

    for (size_t i = 0; i != segments.size(); i += 1)
    {
        for (size_t j = i + 1; j != segments.size(); j += 1)
        {
            if (segments[j].x1 > segments[i].x2) break;

            if (is_intersect(segments[i], segments[j]))
                results.push_back(intersection_t(segments[i], segments[j]));
        }
    }
}

Now computational complexity is reduced to O(N*M), where M is some constant dependent on input data ("how many segments are overlapping by X coordinate"). So in best case complexity is O(N) now (if no segments are overlapping). In worst case complexity is still O(N^2 / 2).

2.2. Bounding Box

Precise calculation of is_intersect predicate is computationally hard (contains considerable amount of multiplication operations, conditional branching, etc). Bounding box is a simple optimization which determines evidently non intersecting segments. Bounding box optimization is based on the following observation - if 2 segments are intersecting in 3D space then their projections to coordinate axises are intersecting by pairs too. Thus, if projections to at least one axis are not intersecting then segments itself are not intersecting in 3D space:

If ((s1.x2 < s2.x1) or (s2.x2 < s1.x1) or

    (s1.y2 < s2.y1) or (s2.y2 < s1.y1) or

     (s1.z2 < s2.z1) or (s2.z2 < s1.z1))

then segments s1 and s2 are not intersecting.
(it's assumed that x1 <= x2, y1 <= y2, z1 <= z2)

Experimentation shows that for randomly generated data bounding box technique detects quite significant amount of evidently non intersecting segments.

Here is the code:

void solve_segment_intersection(std::vector<segment_t>& segments, std::vector<intersection_t>& results)

{

    std::sort(segments.begin(), segments.end(), sort_by_x1);

    for (size_t i = 0; i != segments.size(); i += 1)

    {

        for (size_t j = i + 1; j != segments.size(); j += 4) // note that increment is 4

        {

            if (segments[j].x1 > segments[i].x2)

                break;

            int max_of_mins_y = std::max(segments[i].y1, segments[j].y1);

            int min_of_maxs_y = std::min(segments[i].y2, segments[j].y2);

            if (max_of_mins_y > min_of_maxs_y)

                // evidently no intersection

                continue;

            int max_of_mins_z = std::max(segments[i].z1, segments[j].z1);

            int min_of_maxs_z = std::min(segments[i].z2, segments[j].z2);

            if (max_of_mins_z > min_of_maxs_z)

                // evidently no intersection

                continue;

            // only not calculate precise predicate

            if (is_intersect(segments[i], segments[j]))

                results.push_back(intersection_t(segments[i], segments[j]));

        }

    }

}

2.3. SSE To The Rescue

Bounding box verification can be further optimized with SSE vector operations [3]. First gain comes from the fact that SSE vector operations can process up to 4 pairs of 32-bit integers at a time. Second gain comes from then fact that SSE extensions contain powerful operations that can find minimum/maximum of 2 values in streamlined fashion (w/o conditional branching, just single machine instruction).

Note that in order to apply vector operations data structures have to be converted from AoS (array of structures) representation to SoA (structure of arrays) representation, i.e. following straightforward representation of array of segments:

struct segment_t

{

    int x1, x2, y1, y2, z1, z2;

};

typedef std::vector<segment_t> segments_t;

have to be converted to following SoA representation:

struct segments_t;
{
    std::vector<int> x1, x2, y1, y2, z1, z2;
};

  Normal
  0

  false
  false
  false

  MicrosoftInternetExplorer4

After such conversion we may use SSE vector operations. Here is a bit simplified code (it uses Intel C++ compiler intrinsics):

void solve_segment_intersection(std::vector<segment_t>& segments, std::vector<intersection_t>& results)
{
    std::sort(segments.begin(), segments.end(), sort_by_x1);
    segments_t soa_segments; // population of soa_segments is omitted

    for (size_t i = 0; i != segments.size(); i += 1)
    {
        // load y and z coords of first segment
        __m128i s1_min_y_v = _mm_set1_epi32(soa_segments.y1[i]);
        __m128i s1_max_y_v = _mm_set1_epi32(soa_segments.y2[i]);
        __m128i s1_min_z_v = _mm_set1_epi32(soa_segments.z1[i]);
        __m128i s1_max_z_v = _mm_set1_epi32(soa_segments.z2[i]);

        for (size_t j = i + 1; j != segments.size(); j += 4) // note that increment is 4
        {
            if (segments[j].x1 > segments[i].x2)
                break;

            // load y and z coords of second segment
            __m128i s2_min_y_v = _mm_load_si128((__m128i*)&soa_segments.y1[j]);
            __m128i s2_max_y_v = _mm_load_si128((__m128i*)&soa_segments.y2[j]);
            __m128i s2_min_z_v = _mm_load_si128((__m128i*)&soa_segments.z1[j]);
            __m128i s2_max_z_v = _mm_load_si128((__m128i*)&soa_segments.z2[j]);

            // find bounding box projection to y axis
            __m128i max_of_mins_y = _mm_max_epi32(s1_min_y_v, s2_min_y_v);
            __m128i min_of_maxs_y = _mm_max_epi32(s1_max_y_v, s2_max_y_v);

            // find bounding box projection to z axis
            __m128i max_of_mins_z = _mm_max_epi32(s1_min_z_v, s2_min_z_v);
            __m128i min_of_maxs_z = _mm_max_epi32(s1_max_z_v, s2_max_z_v);

            // check whether segments overlap by y and z coords
            __m128i cmp_y = _mm_cmpgt_epi32(max_of_mins_y, min_of_maxs_y);
            __m128i cmp_z = _mm_cmpgt_epi32(max_of_mins_z, min_of_maxs_z);

            // aggregate results for y and z axises
            __m128i cmp_yz = _mm_or_si128(cmp_y, cmp_z);

            if (_mm_test_all_ones(cmp_yz))
                // neither of these segments are intersecting
                continue;

            if (0 == _mm_extract_epi32(cmp_yz, 0))
                // bounding box says that these segments are possibly intersecting
                // so make precise verification
                if (is_intersect(segments[i], segments[j+0]))
                    results.push_back(intersection_t(segments[i], segments[j+0]));

            // analogously for other segments
            if (0 == _mm_extract_epi32(cmp_yz, 1) && is_intersect(segments[i], segments[j+1]))
                results.push_back(intersection_t(segments[i], segments[j+1]));
            if (0 == _mm_extract_epi32(cmp_yz, 2) && is_intersect(segments[i], segments[j+2]))
                results.push_back(intersection_t(segments[i], segments[j+2]));
            if (0 == _mm_extract_epi32(cmp_yz, 3) && is_intersect(segments[i], segments[j+3]))
                results.push_back(intersection_t(segments[i], segments[j+3]));
        }
    }
}

Note that there is only a handful of lightweight instructions (no multiplications, divisions, branching) before final _mm_test_all_ones() test which throws away 4 segments at a time (assumed to be common case for randomly generated data). And only if the test fails we consider each pair of segments individually. Is_intersecting predicate will be computed only for segment pairs for which bounding box says that segments are possibly intersecting.

3. Parallelization

3.1. Parallelization Outline

As a tool for parallelization I choose Intel Threading Building Blocks (TBB) library which provides handy and flexible abstraction of lightweight tasks. Here is the high-level scheme of parallelization (each rectangle represents a task, rectangles situated on the same horizontal level may be executed in parallel):

3.2. Start Phase

On the start phase I analyze command line parameters, read number of segments, decide on number of worker threads and start input tasks.

Number of worker threads is crucial aspect for performance. Too high number of threads will introduce unjustified latency for small inputs, because of the overheads related to worker thread starting/stopping, work distribution, aggregation of results and synchronization. I've made a number of tests and set up minimum number of segments per worker thread to 1000. I.e. if there is less than 1000 segments all work will be performed by single thread, if there is 1000-2000 segments all work will be performed by 2 worker threads, and so on. I choose number 1000 based on the following equation. Execution time for 1000 segments for 1 thread is roughly equal to that for 2 threads, i.e. at this point parallelization related speedup starts outweighing parallelization related overheads.

Maximum number of worker threads is bounded by number of available execution units (processors, cores, HT threads). Higher number of threads is senseless for CPU-bound tasks (as opposed to IO-bound tasks), because will only cause additional overheads related to context switching.

3.3. Input Phase

Each input task is supplied with it's own piece of input file (input file is equally partitioned between tasks). First of all input task finds nearest to the piece begin boundary of segment description (using '\r' and '\n' symbols as markers). Then it parses input file until piece end, stores segment descriptions to array and collects some statistics.

3.4. Sort Phase

On sorting phase I just use TBB's standard tbb::parallel_sort() algorithm. As will be seen in the Performance section it achieves linear scalability and provides single-threaded performance similar to that of std::sort() algorithm. So no need to re-invent the wheel.

3.5. Intersection Phase

In order to parallelize intersection testing segment pairs must be somehow partitioned to independent groups. Since we have 2 nested loops - outer "i" loop and inner "j" loop - and all iterations of both loops are independent (no data dependencies between them) we may choose either of them as a source of partitioning. However golden rule of parallelization says:

Choose highest possible level for parallelization.

Parallelization on highest possible level tends to "distribute" threads from each other and give each thread bigger piece of independent work, thus reducing work distribution and synchronization overheads.

Parallelization of outer loop may be infeasible/impossible for some reasons. For example, if there is just too small number of iterations (less than number of threads), or if there are data dependencies between iterations (calculations on i+1 iteration depend on results of i-th iteration). However it's not the case for our problem, so I choose partitioning of outer "i" loop.

Initially I created one task per worker thread and equally divided iteration space among them, however it turns out that this way amount of work per task (thread) may be quite unbalanced. If segments are sorted by X coordinate ascending then first segment must be verified for intersection potentially with all other segments, while last segment must not be verified at all.

In order to overcome this I create larger number of tasks which is determined by the following formula:

number_of_tasks = min(number_of_segments / min_number_of_segments_per_task, number_of_worker_threads * surplus_factor);

min_number_of_segments_per_task is set to 1000 (see section 3.2. above).

surplus_factor is set to 64 and determines number of tasks per worker thread required to achieve sufficiently good load balancing.

3.5. Output Phase

I decide to not parallelize output phase because number of intersections is expected to be small.

However if number of intersections is expected to be very large following parallelization strategy can be applied. Memory mapping of sufficient size is opened for output file. Private buffer able to hold, let's say, 1000 intersections is allocated for every worker thread. Worker threads collect results in these private buffers. When a thread's buffer becomes full, thread reserves a range of output file with single atomic increment operation on file_end_marker variable, and then flushes private buffer to file. When thread finishes work it has to flush it's partially full private buffer to file too. This way each thread will output it's own hot-in-cache results, and output will be evenly distributed and parallelized.

4. Performance

I used 2 different data generation algorithms for performance testing. First algorithm generates large segments with large coordinates domain. Here is the results I got on Intel Core2 Duo P9500 (2.5GHz, 6MB L2 cache, 1GHz memory bus) for 30'000 segments with coordinates randomly generated in the range 0 ... 10^6 (number of intersections is 0):

Second algorithm generates small segments with small coordinates domain. Here is the results I got for 400'000 segments of maximum length 40 (in each direction), all coordinates are in range 0...400 (~13K intersections):

As on can see, parallelized phases achieve nearly linear speedup. There is some fixed deviation from the linear scaling probably caused by thread management, work distribution and synchronization overheads. Execution time of non-parallelized output phase is negligible. So I consider parallelization as successful in whole.

References

[1] Bentley-Ottmann algorithm. http://en.wikipedia.org/wiki/Bentley%E2%80%93Ottmann_algorithm

[2] Linear-Time Algorithms for Geometric Graphs with Sublinearly Many Crossings. http://www.siam.org/proceedings/soda/2009/SODA09_018_eppsteind.pdf

[3] http://en.wikipedia.org/wiki/Streaming_SIMD_Extensions

As Chris Thomasson correctly noted, the XCHG-based producer part can be combined with the well-known CAS-based consumer part in order to get multi-producer/multi-consumer (MPMC) queue:
http://groups.google.ru/group/comp.programming.threads/browse_frm/thread/053e322ea90e4ad5

In general, one may combine different producer and consumer parts into single queue provided that queue structure stays the same. For example, it's possible to combine the XCHG-based producer part with consumer part from 2-LOCK queue algorithm. The resulting 1-LOCK/1-XCHG queue will have quite appealing characteristics (wait-free producers, and 1 spin-lock acquisition for consumers, no need for ABA prevention nor safe memory reclamation).

But what I'm going to show is a bit more interesting, it's a novel consumer part for MPMC queue.
The problem with classical CAS-based MPMC queue is that both producers and consumers may touch potentially freed/reused memory, moreover producers may write to that memory. That's why it requires safe memory reclamation (SMR), and in general SMR is quite problematic in a non-managed non-kernel environment (C/C++).
XCHG-based producer part gracefully avoids touching freed/reused memory. So now the problem is with consumer part only, but note that consumers may only read from freed/reused (no writes to that memory). The key point of the proposed algorithm is handling of reads from reused memory with failing CAS, and handling of reads from freed memory with SEH/signal handler.
Main characteristics of the algorithm:
- intrusive
- producers: 1 XCHG, wait-free
- consumers: 1 CAS on common path, mostly lock-free (***)
- producers and consumers do not content with each other (until queue is empty)
- no need for safe memory reclamation

(***) requires additional comments. There is a small (1 machine instruction in length) window of inconsistency for producers. If producer will be preempted there he may (or may not) cause blocking of consumers (other producers are still wait-free). If producer will be terminated there he will cause system-wide stall. Taking into account length of the window, probability of these things may be considered negligible in most situations.

The algorithm requires double-word CAS (for pointer + ABA counter). On 64-bit systems it may be reduced to single-word (64-bit) CAS with pointer packing technique. For example, on Intel64/Windows any aligned pointer may be packed to 39 bits, this allows for 25-bit ABA counter.

OK, here we go:

/*  Multi-producer/multi-consumer queue

 *  2009, Dmitriy V'yukov

 *  Distributed under the terms of the GNU General Public License

 *  as published by the Free Software Foundation,

 *  either version 3 of the License,

 *  or (at your option) any later version.

 *  See: http://www.gnu.org/licenses

 */

// 32-bit, Windows, MSVC

#include <windows.h>

#include <intrin.h>

class mpmc_queue

{

public:

    struct node_t

    {

        node_t* volatile        next_;

    };

    mpmc_queue()

    {

        head_.ptr_ = 0;

        head_.cnt_ = 0;

        tail_ = &head_.ptr_;

    }

    ~mpmc_queue()

    {

        ASSERT(head_.ptr_ == 0);

        ASSERT(tail_ == &head_.ptr_);

    }

    void enqueue(node_t* node)

    {

        ASSERT(node);

        node->next_ = 0;

        node_t** prev = (node_t**)

            _InterlockedExchange((long*)&tail_, (long)node);

        ASSERT(prev);

        // <--- the window of inconsistency is HERE (***)

        prev[0] = node;

    }

    node_t* dequeue()

    {

        unsigned retry_count = 0;

        retry:

        __try

        {

            head_t h;

            h.ptr_= head_.ptr_;

            h.cnt_ = head_.cnt_;

            for (;;)

            {

                node_t* n = h.ptr_;

                if (n == 0)

                    return 0;

                if (n->next_)

                {

                    head_t xchg = {n->next_, h.cnt_ + 1};

                    __int64 prev_raw =

                        _InterlockedCompareExchange64

                            (&head_.whole_, xchg.whole_, h.whole_);

                    head_t prev = *(head_t*)&prev_raw;

                    if (*(__int64*)&prev == *(__int64*)&h)

                        return n;

                    h.ptr_ = prev.ptr_;

                    h.cnt_ = prev.cnt_;

                }

                else

                {

                    node_t* t = (node_t*)tail_;

                    if (n != t)

                    {

                        // spinning here may only be caused

                        // by producer preempted in (***)

                        SwitchToThread();

                        h.ptr_= head_.ptr_;

                        h.cnt_ = head_.cnt_;

                        continue;

                    }

                    head_t xchg = {0, h.cnt_ + 1};

                    head_t prev;

                    prev.whole_ = _InterlockedCompareExchange64

                        (&head_.whole_, xchg.whole_, h.whole_);

                    if (prev.whole_ == h.whole_)

                    {

                        node_t* prev_tail = (node_t*)

                         _InterlockedCompareExchange

                         ((long*)&tail_, (long)&head_.ptr_, (long)n);

                        if (prev_tail == n)

                            return n;

                        // spinning here may only be caused

                        // by producer preempted in (***)

                        while (n->next_ == 0)

                            SwitchToThread();

                        head_.ptr_ = n->next_;

                        return n;

                    }

                    h.ptr_ = prev.ptr_;

                    h.cnt_ = prev.cnt_;

                }

            }

        }

        __except ((GetExceptionCode() == EXCEPTION_ACCESS_VIOLATION

                && ++retry_count < 64*1024) ?

                EXCEPTION_EXECUTE_HANDLER : EXCEPTION_CONTINUE_SEARCH)

        {

            goto retry;

        }

    }

private:

    union head_t

    {

        struct

        {

            node_t*             ptr_;

            unsigned            cnt_;

        };

        __int64                 whole_;

    };

    head_t volatile             head_;

    char                        pad_ [64];

    node_t* volatile* volatile  tail_;

    mpmc_queue(mpmc_queue const&);

    mpmc_queue& operator = (mpmc_queue const&);

};

And here is a small test:

/*  Multi-producer/multi-consumer queue

 *  2009, Dmitriy V'yukov

 *  Distributed under the terms of the GNU General Public License

 *  as published by the Free Software Foundation,

 *  either version 3 of the License,

 *  or (at your option) any later version.

 *  See: http://www.gnu.org/licenses

 */

size_t const thread_count = 8;

size_t const batch_size = 32;

size_t const iter_count = 400000;

bool volatile g_start = 0;

struct my_node : mpmc_queue::node_t

{

    int data;

    char pad [64];

};

unsigned __stdcall thread_func(void* ctx)

{

    mpmc_queue& queue = *(mpmc_queue*)ctx;

    srand((unsigned)time(0) + GetCurrentThreadId());

    size_t pause = rand() % 1000;

    my_node* node_cache [batch_size];

    for (size_t i = 0; i != batch_size; i += 1)

    {

        node_cache[i] = new my_node;

        node_cache[i]->data = i;

    }

    while (g_start == 0)

        SwitchToThread();

    for (size_t i = 0; i != pause; i += 1)

        _mm_pause();

    for (int iter = 0; iter != iter_count; ++iter)

    {

        for (size_t i = 0; i != batch_size; i += 1)

        {

            queue.enqueue(node_cache[i]);

        }

        for (size_t i = 0; i != batch_size; i += 1)

        {

            for (;;)

            {

                my_node* node = (my_node*)queue.dequeue();

                if (node)

                {

                    node_cache[i] = node;

                    break;

                }

                SwitchToThread();

            }

        }

    }

    return 0;

}

int main()

{

    mpmc_queue queue;

    HANDLE threads [thread_count];

    for (int i = 0; i != thread_count; ++i)

    {

        threads[i] = (HANDLE)_beginthreadex

              (0, 0, thread_func, &queue, 0, 0);

    }

    Sleep(1);

    unsigned __int64 start = __rdtsc();

    g_start = 1;

    WaitForMultipleObjects(thread_count, threads, 1, INFINITE);

    unsigned __int64 end = __rdtsc();

    unsigned __int64 time = end - start;

    std::cout << "cycles/op=" << time /

        batch_size * iter_count * 2 * thread_count)

        << std::endl;

}

Here you may download complete Microsoft Visual Studio solution:
mpmc_seh_queue.zip

Radix Sort

Radix sort is a sorting algorithm that sorts integers by processing individual digits. Because integers can represent strings of characters and specially formatted floating point numbers, radix sort is not limited to integers. Most digital computers internally represent all of their data as electronic representations of binary numbers, so processing the digits of integer representations by groups of binary digit representations is most convenient. Two classifications of radix sorts are least significant digit (LSD) radix sorts and most significant digit (MSD) radix sorts. LSD radix sorts process the integer representations starting from the least significant digit and move towards the most significant digit. MSD radix sorts work the other way around. MSD sorting algorithm has particular application to parallel computing, as each of the subdivisions can be sorted independently of the rest.

Radix sort is not a comparison-based sort, so theoretical limit of O(NlgN) is not applicable. Computational complexity of radix sort is O(NK), where N is the number of values and K is the number of subdivisions. This complexity holds for worst, best and mean cases. Space complexity is O(NK).

Single-Threaded Implementation

Naïve single-threaded implementation of MSD radix sort is quite straightforward:

typedef std::vector<std::vector<byte> > data_t;
size_t const byte_values = 256;
typedef unsigned char byte;
void radix_sort(data_t& data, size_t position = 0)
{
    // recursion stop conditions
    if (data.size() <= 1 || position == data[0].size())
        return;
    std::vector<data_t> radix (byte_values);
    // radix split
    for (size_t i = 0; i != data.size(); ++i)
    {
        size_t idx = data[i][position];
        radix[idx].push_back(data[i]);
    }
    size_t out_pos = 0;
    for (size_t i = 0; i != byte_values; ++i)
    {
        // recursive sort of lesser significant digits
        radix_sort(radix[i], position + 1);
        // copyback
        for (size_t j = 0; j != radix[i].size(); ++j, ++out_pos)
        {
            data[out_pos] = radix[i][j];
        }
    }
}

Parallelization

I use 2 types of parallelization. First type is the parallelization of the radix split (intra-radix parallelization), this parallelization is especially useful for initial radix split (most significant digit). Input data is split into several parts (fork), each processor picks up a part and makes radix split (parallel processing). When all parts have split partial radix arrays are aggregated (join) and directed to the next level of recursion. This parallelization may help also with sorting of not-so-randomly distributed data.

Second type is the parallelization on inter-radix level. Processor completely sorts whole array on lower levels of recursion. This parallelization helps mitigate overheads of thread synchronization.

Parallelization is guided at run-time. I.e. threads prefer to do inter-radix parallelization, however if some threads are out of work they help other threads on intra-radix level.

When size of the input array reaches some threshold, thread switches to single-threaded mode, i.e. no further sub-tasks are split (this also helps mitigate synchronization overheads).

Here is pseudo-code of the parallel algorithm:

struct radix_desc

{

  // partial results

  std::vector<data_t>     radix [thread_count];

  size_t                  radix_pending_count;

  size_t                  position;

  //...

};

struct radix_task

{

  data_t                  input;

  radix_desc&             desc;

  //...

  void execute()

  {

    // partial radix split

    for (size_t i = 0; i != input.size(); ++i)

    {

      size_t idx = input[i][desc.position];

      desc.radix[thread_id][idx].push_back(input[i]);

    }

    if (0 == atomic_decrement(desc.radix_pending_count))

    {

      // spawn sub-tasks

      for (size_t i = 0; i != byte_values; ++i)

      {

        // aggregate partial results

        data_t result;

        for (size_t j = 0; j != thread_count; ++j)

        {

          result.insert(result.end(), desc.radix[j][i]);

        }

        radix_desc desc = new radix_desc (...);

        spawn_some_subtasks(desc, result);

      }

    }

  }

};

Scheduling

I've implemented custom task-based scheduler on top of the Win32 threading API. In main part it's similar to classical Cilk-style work-stealing scheduler, though I've made some improvements on it. In particular I've added system-topology awareness, hyper-threading awareness, affinity-awareness, batch-spawn capability and manual task-depth control. All worker threads are strictly binded to EUs (execution units), stealing conducted based on the “distance” between EUs, i.e. worker thread tries to steal from neighbor threads first, then from threads running on different NUMA node (system-topology awareness). This allows to efficiently reuse data in shared L3 cache of the processors.

Sibling HT threads share single work-stealing deque (HT awareness), this allows them to keep as close to each other as possible in terms of working sets. Resources of single core (L1D cache, L1 DTLB, etc) are not capable to accommodate 2 distinct radix sorts, HT awareness allows HT sibling threads to work on single radix sort, so to say. Assume first HT thread completes radix split and spawns a bunch of sub-tasks. Then it picks up some sub-task to process, while second HT thread picks up another sub-task, data for that another sub task is already in L1D cache (as well as in L1 DTLB) of the core.

The scheduler is able to support affinity of tasks. Though I didn't have enough time to exploit the feature.

When thread completes radix split it submits up to 96 (number of printable characters in US-ASCII) sub-tasks, scheduler allows to submit all the tasks in single enqueue operation. This reduces synchronization overheads to some degree.

When thread submits new tasks to the scheduler it explicitly passes so called tasks depth as a parameter. Task depth relates to the task level in the work DAG. When thread pops task from own work-stealing deque it picks up task with the highest available level (the smallest piece of work), when thread steals task from remote work-stealing deque it picks up task the lowest available level (the biggest piece of work). This reduces number of steal operations.

Regarding Threading Building Blocks. Another possibility would be to use TBB's task scheduler. Usage of the TBB would not affect main logic of the program in any way, because it supports exactly the same task concept. On one hand TBB would allow to reduce amount of written code (no need to implement scheduler manually). On the other hand TBB's scheduler is not system-topology aware, not HT aware, does not provide batch spawn capability, and does not provide manual control over task depths (not relevant w/o HT awareness) (TBB's scheduler is affinity aware to some degree, i.e. it supports task affinities however does not supports thread affinities). Also TBB's scheduler has somehow bigger task spawn/consume overheads: some 600 cycles, while my scheduler some 200 cycles (on my hardware). Since the contest is about raw performance I've decided to implement own scheduler.

Single-threaded Optimizations

Avoiding copyback. Naïve radix sort implementation makes K (number of digits) copies of the whole data set in the copyback phase. In order to eliminate those copies I use following optimization. On start I allocate array for the sorted data:

struct output_cell

{

  int count_;

  uint32_t*   data_;

};

size_t const output_size = 96*128*128; 

output_cell* g_output = new output_cell [output_size];

3 most significant digits of the value determine index in that array:

size_t output_index(uint64_t val)

{

  byte* v = (byte*)&val;

  return ((size_t)v[3]) | ((size_t)v[2] << 7) | (((size_t)v[1] - 32) << 14);

}

4 least significant digits of the value are stored in the inner array:

void store_result(uint64_t val, size_t position)

{

  size_t idx = output_index(val);

  uint32_t v = (uint32_t)(val >> 32);

  g_output[idx].data_[position] = v;

}

This way all copies of the data in the copyback phase are eliminated, sorted data are placed directly to the final destination.

Counting sort. When values reduced to 2 bytes (by 5 previous radix splits) I use counting sort (which is a special case of the radix sort with special intermediate representation of the values). Counting sort has the same computational complexity as the radix sort, however has lower space complexity and can be implemented more efficiently. Since I expect very few values will be sorted with counting sort at a time (i.e. counter array will be very sparse), I add bitmask to optimize search over counter array.

Pseudo-code of the counting sort:

void counting_sort(uint16_t* begin, uint16_t* end, uint32_t* output, uint32_t prefix)

{

  uint32_t counter [256*256] = {};

  bitmask_t bitmask;

  for (uint16_t* pos = begin; pos != end; pos += 1)

  {

    uint16_t v = pos[0];

    counter[v] += 1;

    bitmask.set_bit(v);

  }

  for (uint16_t v; bitmask.get_and_reset_bit(v);)

  {

    do

    {

      uint32_t val = prefix;

      val |= v;

      output[0] = val;

      output += 1;

    }

    while (--counter[v]);

  }

}

bitmask_t::get_and_reset_bit() operation is implemented with the BSF instruction (_BitScanForward64() intrinsic). Bitmask optimization reduces computational complexity of the counting sort from 65536*N to 2*N.

Counting sort is not parallelized in my implementation. Since input data is uniformly distributed, I expect this to not affect performance. Though this is a possible further optimization which will allow better handling of not-so-randomly distributed data.

Template code generation. I heavily use C++ template programming in order to allow efficient code generation. Value is represented by the following class:

template<size_t digits_t> struct data_layout;

template<> struct data_layout<7> {typedef uint64_t value_t;};

template<> struct data_layout<6> {typedef uint64_t value_t;};

template<> struct data_layout<5> {typedef uint64_t value_t;};

template<> struct data_layout<4> {typedef uint32_t value_t;};

template<> struct data_layout<3> {typedef uint32_t value_t;};

template<> struct data_layout<2> {typedef uint16_t value_t;};

template<size_t digits_t>

struct value

{

  typedef typename data_layout<digits_t>::value_t value_t;

  value_t val;

  value& operator = (value<digits_t + 1> const& r)

  {

    val = (value_t)(r.val >> (8 * (sizeof(r) - sizeof(*this))));

    return *this;

  }

  char prefix() const

  {

    return ((char*)&val)[sizeof(*this) - digits_t];

  }

};

All functions and classes related to radix sorting are also template parametrized by number of digits, and act accordingly to particular value layout, location of the radix prefix in the value, etc.

Also radix task is template parametrized by parameters is_single_threaded and is_parent_single_threaded. When is_single_threaded==true, task allocates subtasks on the stack and executes them directly. When is_parent_single_threaded==true, task avoids atomic counting of pending siblings, since parent allocates sub-tasks on the stack they all will complete when parent completes.

Memory allocation. Efficient memory allocation is crucial for single-threaded as well as multi-threaded (standard Windows allocator uses single mutex which significantly reduces scalability) performance of the implementation. I implement distributed region memory allocator, there is a pool of 2 MB pages per NUMA node, a thread privatizes a page from that pool and then uses region allocation on the page. When page exhausted thread privatizes another page, and so on. No memory is freed to the OS during radix sort, though some memory is reused internally. Also I implement simple caching memory allocator for objects of a particular size; the allocator is based on a per-thread lifo freelist. When object is freed it’s pushed onto the freelist, when object must be allocated it’s popped from the freelist.

Tools

I was considering Microsoft Visual C++ (MSVC) and Intel C++ (ICC) compilers. In 32-bit mode ICC showed impressive 30% speedup over MSVC (even more with profile-guided optimizations). However in 64-bit mode ICC showed wicked 20% slowdown (with maximum possible optimizations turned on, including /QxHost, /Qunroll, etc), profile-guided optimizations improve situation somehow but ICC still was behind MSVC. I didn't have time to investigate the problem, so I've decided to use MSVC for final submission.

As a profiler I used AMD CodeAnalyst, it's a simple profiler which allows to easily capture and analyze profile of the program. Profiling was crucial for single-threaded optimizations. Also it allowed me to verify that profile of the multi-threaded version is mainly identical to that of the single-threaded version, and that overheads for synchronization and scheduling are not greater than several percents – all this is a good sign of successful parallelization. Another option would be to use Intel PTU, it's somehow more complicated however would allow to capture processor performance events which is crucial for single-threaded optimization (for example it would answer what causes excessive pipeline stalls – L1D cache misses or L1 DTLB misses).

Another great tool I used is Windows Task Manager. I allowed me to track virtual memory consumption, CPU utilization, working set and number of page faults. The goal was to keep virtual memory consumption in expected bounds (~1.5 * input data size in my case), 100% utilization of the CPUs in parallel phase and 0 page faults (i.e. working set == virtual memory).

]]> http://software.intel.com/en-us/blogs/2009/05/06/another-sorts-of-sorts/feed/ 0 http://software.intel.com/en-us/blogs/2009/03/03/dont-rely-on-memory-barriers-for-synchronization-only-if-you-dont-aware-of-relacy-race-detector/ http://software.intel.com/en-us/blogs/2009/03/03/dont-rely-on-memory-barriers-for-synchronization-only-if-you-dont-aware-of-relacy-race-detector/#comments Tue, 03 Mar 2009 18:23:26 +0000 Dmitriy Vyukov http://software.intel.com/en-us/blogs/2009/03/03/dont-rely-on-memory-barriers-for-synchronization-only-if-you-dont-aware-of-relacy-race-detector/ Multithreading is hard. Implementation of synchronization primitives is even harder. And most advanced synchronization primitives which exploit relaxed memory models are brain damaging.  So don't rely on memory barriers for synchronization. Now you will think "Ah, you are one of those folks, who constantly saying to us - don't do this, don't do that, it's too difficult, it's too dangerous, anyway you will fail". Nope, I am not. I am saying exactly the opposite - do rely on memory barriers for synchronization... sometimes; because smart fine-grained synchronization is a matter of orders of magnitude performance difference and principal possibly of scaling (again sometimes, of course). But it's true that low-level synchronization primitives and memory ordering issues are hard. So some time ago I've developed a tool called Relacy Race Detector in order to help developers (actually in order to help me, but it doesn't matter) with these issues.

Relacy Race Detector (RRD) is a synchronization algorithm verifier for relaxed memory models (formally - stateless dynamic verifier, if you are interested). Physically it's a header-only C++ library, but it's able to test not only C++ algorithms, but also Java, .NET/CLI, x86, PowerPC, SPARC, etc - more on this later. Now let's consider how work with RRD looks for end user.

First of all you have to implement synchronization algorithm (mutex, concurrent data structure, etc) which is subject to verification. Then you express one or several unit-tests of the algorithm. Then you write several lines of code to start execution of the unit-test. Ok, now you may compile and start the program, and RRD will take care of the efficient execution of the unit-test, i.e. it will execute zillions of different thread interleavings. During execution of each interleaving it will be constantly conducting a number of built-in checks - checks for data races, dead-locks, live-locks, accesses to freed memory, double memory free, memory leaks, accesses to uninitialized variables, incorrect API usage (recursive lock on non-recursive mutex), as well as verification of user specified asserts and invariants. When (if) RRD will detect some error it will output detailed execution history that leads to the error (history includes such things as instances of the ABA problem, reorderings of memory accesses, thread blocking/unblocking). With such history in hands localization of the error is less than a problem.

Great number of various built-in checks allows to specify no user asserts/invariants in many cases and still get exhaustive verification; for example, most mutual exclusion algorithms may be verified w/o user asserts. However, if you want to verify FIFO order of messages provided by producer-consumer queue, you will have to code this check manually.

Regarding verification of Java/.NET programs. Initially I was targeted only at C++0x memory model and atomics API. But it turns out that C++0x memory model is so relaxed and atomics API is so general that I was easily able to map other memory models to C++0x memory model. So now RRD includes a set of thin wrappers for Java atomics and volatiles, .NET interlocked and volatiles, as well as POSIX and Win32 synchronization primitives. Thus, yes, you still have to code in C++, but you can verify your algorithm implementation as is would Java or .NET algorithm implementation.

RRD contains 3 different schedulers: random scheduler, full search scheduler and context bound scheduler. Each represents some compromise between verification speed and completeness of the verification. All schedulers are fair, i.e. support verification of formally non-terminating programs. I don't want to go into more details regarding schedulers here, but you can find more info on Relacy Race Detector web site.

Now let's look at the simple example which mimics basic spin-mutex:

#include <relacy/relacy.hpp>
// main RRD namespace is 'rl', also note required instrumentation in the form of '($)'

struct mutex {
  rl::atomic<int> lock_;
   mutex() {
     lock_($) = 0;
   }
   void lock() {
     while (lock_($).exchange(1, rl::memory_order_acquire) == 1)
       rl::yield($, 1);
  }
  void unlock() {
     lock_($).store(0, rl::memory_order_release);
   }
};

// unit-test ('2' means number of threads)
struct mutex_test : rl::test_suite<mutex_test, 2> {
  mutex mtx;
  rl::var<int> data;
  // executed in single thread before main thread function
  void before() {
    data($) = 0;
  }
  // main thread function
  void thread(unsigned /*thread_index*/) {
    mtx.lock();
    data($) += 1;
    mtx.unlock();
  }
  // executed in single thread after main thread function
  void after() {
    RL_ASSERT(data($) == 2);
  }
};

int main() {
  rl::simulate<mutex_test>();
}

If we will run it we will see successful completion of the test. Now let's introduce some simple bugs into the code and see how RRD will react on them. I've replaced rl::memory_order_release with rl::memory_order_relaxed in mutex::unlock() and run the program, the output is (I've omitted per-thread execution history and some non relevant details for brevity; [first number] in history is global operation index, second number is thread index):

struct mutex_test
 DATA RACE (data race detected)
 iteration: 1

execution history:
 [0] 1: [CTOR BEGIN]
 [1] 1: <0034BEE0> atomic store, value=0, (prev value=0), order=seq_cst, in mutex::mutex, mutex.cpp(7)
 [2] 1: <0034BF04> store, value=0, in mutex_test::mutex_test, mutex.cpp(22)
 [3] 1: [CTOR END]
 [4] 1: <0034BEE0> exchange , prev=0, op=1, new=1, order=acquire, in mutex::lock, mutex.cpp(10)
 [5] 1: <0034BF04> load, value=0, in mutex_test::thread, mutex.cpp(26)
 [6] 1: <0034BF04> store, value=1, in mutex_test::thread, mutex.cpp(26)
 [7] 1: <0034BEE0> atomic store, value=0, (prev value=1), order=relaxed, in mutex::unlock, mutex.cpp(14)
 [8] 0: <0034BEE0> exchange , prev=0, op=1, new=1, order=acquire, in mutex::lock, mutex.cpp(10)
 [9] 0: <0034BF04> load, value=0, in mutex_test::thread, mutex.cpp(26)
 [10] 0: DATA RACE (data race detected), in mutex_test::thread, mutex.cpp(26)

We see that RRD easily detected data race on variable mutex_test::data (more precisely this is data race type 2 which means that conflicting data accesses are not adjacent in the execution history although still conflicting).
Now let's make following change in the test:

struct mutex_test : rl::test_suite<mutex_test, 2> {
  mutex mtx;
  rl::atomic<int> data;
  void before() {
    data($) = 0;
  }
  void thread(unsigned /*thread_index*/) {
    mtx.lock();
    int tmp = data($).load(rl::memory_order_relaxed);
    data($).store(tmp + 1, rl::memory_order_relaxed);
    mtx.unlock();
  }
  void after() {
    RL_ASSERT(data($) == 2);
  }
};

Now mutex_test::data is declared as 'rl::atomic<int>' (so no more data races are possible on it), and I've slightly changed the way I increment the variable. Here is the output (remember, unlock operation is still done with memory_order_relaxed):

struct mutex_test
USER ASSERT FAILED (assertion: data($) == 2)
iteration: 5

execution history:
[0] 1: [CTOR BEGIN]
[1] 1: <0034AD10> atomic store, value=0, (prev value=0), order=seq_cst, in mutex::mutex, mutex.cpp(7)
[2] 1: <0034AD34> atomic store, value=0, (prev value=0), order=seq_cst, in mutex_test::mutex_test, mutex.cpp(22)
[3] 1: [CTOR END]
[4] 1: <0034AD10> exchange , prev=0, op=1, new=1, order=acquire, in mutex::lock, mutex.cpp(10)
[5] 0: <0034AD10> exchange , prev=1, op=1, new=1, order=acquire, in mutex::lock, mutex.cpp(10)
[6] 0: yield(1), in mutex::lock, mutex.cpp(11)
[7] 1: <0034AD34> atomic load, value=0, order=relaxed, in mutex_test::thread, mutex.cpp(26)
[8] 1: <0034AD34> atomic store, value=1, (prev value=0), order=relaxed, in mutex_test::thread, mutex.cpp(27)
[9] 0: <0034AD10> exchange , prev=1, op=1, new=1, order=acquire, in mutex::lock, mutex.cpp(10)
[10] 0: yield(1), in mutex::lock, mutex.cpp(11)
[11] 1: <0034AD10> atomic store, value=0, (prev value=1), order=relaxed, in mutex::unlock, mutex.cpp(14)
[12] 0: <0034AD10> exchange , prev=0, op=1, new=1, order=acquire, in mutex::lock, mutex.cpp(10)
[13] 0: <0034AD34> atomic load, value=0 [NOT CURRENT], order=relaxed, in mutex_test::thread, mutex.cpp(26)
[14] 0: <0034AD34> atomic store, value=1, (prev value=1), order=relaxed, in mutex_test::thread, mutex.cpp(27)
[15] 0: <0034AD10> atomic store, value=0, (prev value=1), order=relaxed, in mutex::unlock, mutex.cpp(14)
[16] 0: [AFTER BEGIN]
[17] 0: <0034AD34> atomic load, value=1, order=seq_cst, in mutex_test::after, mutex.cpp(31)
[18] 0: USER ASSERT FAILED (assertion: data($) == 2), in mutex_test::after, mutex.cpp(31)

We see that RRD have detected on the fifth iteration that thread #0 loaded not current value from the 'data' variable, so that final value of the variable is 1 instead of 2.
Ok, now I hope that you have got brief idea of the Relacy Race Detector. In the conclusion I want to describe some of the features that I am thinking to incorporate into future releases of RRD - thread local storage (with POSIX and Win32 wrappers), UNIX signals and hardware interrupts, partial-order reductions and parallelizatioin of the run-time, persistent checkpointing of the simulation, performance simulation, detection of dead-code. Although current development of RDD is a kind of demand-driven, so I will be happy to hear your comments, suggestions and feedback.

Main Relacy Race Detector web-site is:
http://groups.google.com/group/relacy

Assume we have complicated operation which involves non-trivial modifications of several objects/containers. Assume that exception can be thrown either by memory allocator, or by copy constructor of some object, or just by application logic. In order to provide strong exception safety in such situation we have to manually write code for cancellation of all those modifications. This can be non-trivial error-prone task.

TM already has all necessary machinery for cancellation of arbitrary operations. TM doesn't care about complexity of operations, number of involved objects/containers, it can just instantly cancel anything which happens inside atomic block. Why don't use it?

So the general recipe for strong exception safety with the help of TM: wrap arbitrary operation in atomic block, if the case of error just abort transaction. Hocus-pocus! Your operation obtains strong exception safety at once w/o single line of code!

This can be equally applied to exceptions and errors codes/return values, no matter.

Here is something which I was actually able to model with Intel C++ STM Compiler:

// object which "sometimes" throws exception in copy ctor

bool fail_bad_int = false;

int fail_bad_int_step = 8;

__declspec(tm_callable)

struct bad_int

{

    bad_int(int v = 0)

        : v(v)

    {}

    bad_int& operator = (bad_int r)

    {

        if (fail_bad_int && 0 == --fail_bad_int_step)

            throw 0;

        v = r.v;

        return *this;

    }

    operator int () const

    {

        return v;

    }

    bool operator < (bad_int r) const

    {

        return v < r.v;

    }

    int v;

};

// transactional sort function

template<typename T>

__declspec(tm_callable)

void sort(T* begin, T* end)

{

    T temp;

    size_t n = end - begin;

    if (n < 2)

        return;

    bool swapped = false;

    do

    {

        swapped = false;

        n -= 1;

        for (size_t i = 0; i != n; ++i)

        {

            if (begin[i + 1] < begin[i])

            {

                temp = begin[i];

                begin[i] = begin[i + 1];

                begin[i + 1] = temp;

                swapped = true;

            }

        }

    }

    while (swapped);

}

int main()

{

    std::vector<bad_int> x;

    std::generate_n(std::back_inserter(x), 10, rand);

    std::copy(x.begin(), x.end(),

        std::ostream_iterator<int>(std::cout, " \t"));

    std::cout << std::endl;

    fail_bad_int = true;

    bad_int* begin = &*x.begin();

    bad_int* end = begin + x.size();

    __tm_atomic

    {

        try

        {

            sort(begin, end);

        }

        catch (...)

        {

            __tm_abort;

        }

    }

    std::copy(x.begin(), x.end(),

        std::ostream_iterator<int>(std::cout, " \t"));

    std::cout << std::endl;

}

Output:

41 18467 6334 26500 19169 15724 11478 29358 26962 24464
41 18467 6334 26500 19169 15724 11478 29358 26962 24464

And if I comment __tm_abort statement out, then output is:

41 18467 6334 26500 19169 15724 11478 29358 26962 24464
41 6334 18467 19169 26500 26500 11478 29358 26962 24464

Notice that in latter case array is in some intermediate state. And in former case array is in initial state.

Although one has to understand that this method can incur substantial run-time (space and time) overheads even if exception is not thrown. On the other hand it's extremely simple and not error prone (i.e. you can not forget to cancel some things or cancel incorrectly). So this approach can be used on cold-paths or on initial development stage.

I see that Intel C++ STM Compiler has execution mode called 'serial atomic', which used for serialized transactions which have abort statements. Serial atomic mode can be used to hasten such single-threaded usage of TM. I.e. TM run-time only has to log writes, no need for synchronization, quiescence, verification etc.

What do you think? Is it useful?

The first thing one has to say in that context is:
1. Eliminate sharing. Period. Not false sharing, just sharing. It's sharing that has huge performance penalties. It's sharing that changes linear scalability of your application to super-linear degradation. And believe me, hardware has no means to distinguish false sharing from true sharing. It can't penalize only false sharing, and handle true sharing without any performance penalties.

Second thing one has to say in that context is:
2. Put things that must be close to each other... close to each other. Assume following situation. In order to complete some operation thread has to update variable X and variable Y. If variables are situated far from each other (on different cache lines), then thread has to load (from main memory, or from other processor's cache) 2 cache lines instead of 1 (if variables are situated close to each other). Effectively this situation can be considered the same as false-sharing, because thread places unnecessary work on interconnects, thus degrading performance and scalability.

Points 1 and 2 can be aggregated as:

1+2. Do pay attention to data layout. This was important in the 60's. This is even more important in the multicore era.

Only after that one can also add:

3. Sometimes sharing can show up when you are not expecting it, i.e false sharing. This is important to eliminate false sharing too... etceteras... [insert here contents of False Sharing article]

If one says only point 3, well, it's basically senseless. And sometimes it can even hurt.

Let's consider simple example:

long volatile g_operation_count = 0;

void collect_statistics() {

  InterlockedIncrement(&g_operation_count);
}

What does naive programmer think about it? Hmmm... Let's see... I use "fast" non-blocking interlocked operations. Good!... Hmmm... False sharing. Let's see... Hmmm... Here is no false sharing. Good! So my program fully conforms to recommendations of experts.

Rubbish! It's a dead-slow, completely non-scalable program.

Now let's apply consistent rules to the example. First of all we have to do something like this:

long volatile g_operation_count [MAX_THREAD_COUNT] = {};

void collect_statistics() {

  InterlockedIncrement(&g_operation_count[get_current_thread_id()]);
}

It's good distributed design. When we need aggregate number of operations we just sum up all thread local counters.

Only at this point we can remember about false-sharing and put the final touches to the code:

struct counter_t {
  long volatile count;
  char pad [CACHE_LINE_SIZE - sizeof(long)];
}
counter_t g_operation_count [MAX_THREAD_COUNT] = {};

void collect_statistics() {

  InterlockedIncrement(&g_operation_count[get_current_thread_id()].count);
}

Ok, this distributed version is also fast and scalable. It has linear scalability and can be faster up to 100x on modern multi-core hardware as compared with original version.

So, point 1+2 is a kind of general rule, while point 3 is just a refinement to them.

Why people don't say the whole truth? I don't know. I don't beleive that authors don't aware of the problem. Maybe they think that it's obvious. The practice shows that it's not...

I'm going to devote this blog to multi-threading, multi-core, synchronization algorithms and all other concerned things. Development of lock-free, wait-free, obstruction-free, atomic-free synchronization algorithms is my hobby. Some of my designs you can see here.

Also, I've developed a tool called Relacy Race Detector, which can be of help to developers of synchronization algorithms.

Stay tuned!