In a sequential program, it is not a big problem since the calculation order is exactly specified so the result is predictable and repeatable. The situation is not so clear in parallel programming.

To make the example parallel, I used the parallel_reduce template function from Intel® Threading Building Blocks (Intel® TBB):

std::vector<float> arr( N, 1.0f/(float)N );
float sum = tbb::parallel_reduce( tbb::blocked_range( arr.begin(), arr.end() ), 0.0f,
    []( const tbb::blocked_range& r, float sum ) {
        return std::accumulate( r.begin(), r.end(), sum );
    },
    std::plus<float>() );
std::cout << sum << std::endl;

As in the examples above, the code calculates the sum of N fractions, but it uses multiple processor cores if available. As it is well known, we face a disappointing fact of different results being possible for different orders of calculations. If we run it 10 times and N=1000 we will get something like this:

0.999991
1
0.999999
0.999996
0.999998
0.999998
0.999998
1
0.999997
0.999998

It’s worth mentioning that the result differs from run to run! In spite of the fact that the developer specifies the calculations – when it is calculated in parallel the order of calculation gets out of control.

On the other hand, it is not as bad as all that. Although the OS operates on threads and fills the application with indeterminism, it is still possible to manage the order of calculations. One of the new features of Intel TBB 4.0 is the parallel_deterministic_reduce template algorithm. The algorithm has the same interface as parallel_reduce except that it does not allow you to specify a partitioner. (For parallel_reduce it is possible to pass a partitioner as the last argument.) We will discuss why this restriction exists later. But for now, let’s replace the parallel_reduce with parallel_deterministic_reduce and look at how the result changes:

std::vector<float> arr( N, 1.0f/(float)N );
float sum = tbb::parallel_deterministic_reduce( tbb::blocked_range( arr.begin(), arr.end() ), 0.0f,
    []( const tbb::blocked_range& r, float sum ) {
        return std::accumulate( r.begin(), r.end(), sum );
    },
    std::plus<float>() );
std::cout << sum << std::endl;

Again run it 10 times:

1
1
1
1
1
1
1
1
1
1

The key point here is that the result is the same from run to run.

The sources of non-determinism in parallel_reduce derive from partitioning and body splitting. Let’s consider each of these subjects:

• Partitioning. The simple_partitioner determines exactly how many and which subranges are created. It splits the iteration range until each subrange is smaller than a given grain size. Thus the behavior only depends on the range size and grain size specified by the developer. However, other types of partitioning in Intel TBB are non-deterministic: to improve performance of the algorithms, range splitting provided by these partitioners depends on run-time stealing events, which we cannot predict.

• Body splitting. For performance reasons parallel_reduce minimizes body copies: it splits the body only when consecutive subranges are processed by different threads. Thus body splitting, like “advanced” partitioning, also depends on non-deterministic task stealing.

The example shows that parallel_reduce is really inapplicable for non-associative operations like floating point arithmetic. To achieve a repeatable result from a reduction with non-associative operations parallel_deterministic_reduce has been developed. From the considerations of partitioning (given above), it follows that only the simple_partitioner can be used for parallel_deterministic_reduce; and thus, no choice of an alternative partitioner is possible. Consequently, parallel_deterministic_reduce always challenges us with choosing an appropriate grain size. And smart body splitting has been disabled for the sake of deterministic behavior, so for each subrange a new body is created. This fact complicates the challenge of grain size selection even more: on the one hand, a small grain size increases the number of body copying and overall overhead, but on the other hand, a big grain size may lead to imbalance and underutilization. Fig. 1 shows the relative performance of parallel_deterministic_reduce (simple_partitioner with various grain sizes) in comparison with parallel_reduce (auto_partitioner with default grain size). An appropriate grain size provides the same performance of parallel_deterministic_reduce as parallel_reduce, - but an incorrectly chosen grain size may lead to significant performance degradation, as shown in Fig.1 at the extremes of the grain size axis.

Fig.1. Comparison of parallel_reduce (auto_partitioner) and parallel_deterministic_reduce (simple_partitioner) on Pi calculation example.

To demonstrate the split-join order behavior of parallel_deterministic_reduce, a small example is given with range [0, 20) and grain size = 5, similar to examples for parallel_reduce in the Intel TBB Reference manual:

A tree of subranges

For each right node a new body is created by the body split constructor. The slash marks (/) in the tree show where the body split is performed. Thus, for the current example the parallel_deterministic_reduce will always produce 4 subranges and 4 different bodies associated with them. Each of these subranges may be executed in parallel. When both children of a node finish, the corresponding bodies are merged: the right child body “added” to the left child body (in our examples via the std::plus<float>() binary function).

To conclude, parallel_deterministic_reduce provides a deterministic number and deterministic sizes of subranges, and it exactly defines which pairs of subranges are merged. It’s important to note that a repeatable result obtained with parallel_deterministic_reduce may still be different from that obtained via serial execution. Moreover, the results may be different for various grain sizes, since range splitting depends on the grain size. Also, the algorithm is not targeted to improve the accuracy of calculations. The exact result of 1 in the above example of fraction sum calculation has been obtained by chance. For other examples the algorithm can cause a decrease in accuracy. Overall, parallel_deterministic_reduce is not a replacement to parallel_reduce but an alternative solution for those who need repeatability.

Let’s look at two input devices here, the toggle and the pulse (or as I would have liked to have called them, the switch and the clock). A toggle sends a signal of high or low, toggling between the two states, every time it is “toggled” or flipped. A pulse continually alternates between the high and low states at a given duration. The toggle class is implemented as follows:

class toggle {
    graph& my_graph;
    signal_t state;
    overwrite_node < signal_t > toggle_node;
 public:
    toggle(graph& g) : my_graph(g), state(undefined), toggle_node(g) {}
    toggle(const toggle& src) : my_graph(src.my_graph), state(undefined),
                                toggle_node(src.my_graph) {}
    ~toggle() {}
    // Assignment ignored
    toggle& operator=(const toggle& src) { return *this; }
    sender < signal_t > & get_out() { return toggle_node; }
    void flip() {
        if (state==high) state = low;
        else state = high;
        toggle_node.try_put(state);
    }
    void activate() {
        state = low;
        toggle_node.try_put(state);
    }
};

The toggle is represented internally by an overwrite_node, because it simply needs to keep track of one most-recent state. As an input device, it doesn’t receive output from any other items, so it has no explicit input ports, only actions (flip, activate) which can alter the output state. The output port can of course be acquired via get_out, so that the toggle can be used to send signals into a circuit.

The pulse class is a little more interesting:

class pulse {
    class clock_body {
        size_t& ms;
        int& reps;
        signal_t val;
    public:
        clock_body(size_t& _ms, int& _reps) : ms(_ms), reps(_reps), val(low) {}
        bool operator()(signal_t& out) {
            rt_sleep(ms);  // our own portable sleep function
            if (reps>0) --reps;
            if (val==low) val = high;
            else val = low;
            out = val;
            return reps>0 || reps == -1;
        }
    };
    graph& my_graph;
    size_t ms, init_ms;
    int reps, init_reps;
    source_node < signal_t > clock_node;

public:
    pulse(graph& g, size_t _ms=1000, int _reps=-1) :
        my_graph(g), ms(_ms), init_ms(_ms), reps(_reps), init_reps(_reps),
        clock_node(g, clock_body(ms, reps), false)
    {}
    pulse(const pulse& src) :
        my_graph(src.my_graph), ms(src.init_ms), init_ms(src.init_ms),
        reps(src.init_reps), init_reps(src.init_reps),
        clock_node(src.my_graph, clock_body(ms, reps), false)
    {}
    ~pulse() {}
    pulse& operator=(const pulse& src) {
        ms = src.ms; init_ms = src.init_ms;
        reps = src.reps; init_reps = src.init_reps;
        return *this;
    }
    sender < signal_t > & get_out() { return clock_node; }
    void activate() { clock_node.activate(); }
    void reset() { reps = init_reps; }
};

This class is based on the source_node. It generates a signal, alternating between low and high, every ms milliseconds. There is also an option to repeat the alternation a certain number of times and then stop, which is useful for designing simulations that use a clock but also terminate. The source_node body sleeps for a duration before flipping the signal and sending it. It doesn’t begin sending signals immediately, but requires activation. In the case of a non-infinite clock (reps is set), once the pulse object has run for the given number of repetitions, it can be reset and reactivated to use it again.

Next, we discuss two output devices, the LED and the digit. The LED is simply a tiny light that is on while the signal it is receiving is high, and off when the signal is low. For simple text display, the LED looks like this: (*) when it is on and ( ) when it is off. The digit device receives a four-bit input and displays a single hexadecimal digit. For simulations, both devices have the option of continuously displaying their state as it changes, or a silent mode, which displays only when a display method is called.

class led {
    class led_body {
        signal_t &state;
        string &label;
        bool report_changes;
        bool touched;
    public:
        led_body(signal_t &s, string &l, bool r) :
            state(s), label(l), report_changes(r), touched(false)
        {}
        continue_msg operator()(signal_t b) {
            if (!touched || b!=state) {
                state = b;
                if (state != undefined && report_changes) {
                    if (state) printf("%s: (*)\n", label.c_str());
                    else printf("%s: ( )\n", label.c_str());
                }
                touched = false;
            }
            return continue_msg();
        }
    };
    graph& my_graph;
    string label;
    signal_t state;
    bool report_changes;
    function_node < signal_t, continue_msg > led_node;
 public:
    led(graph& g, string l, bool rc=false) : my_graph(g), label(l), state(undefined),
        report_changes(rc), led_node(g, 1, led_body(state, label, report_changes))
    {}
    led(const led& src) : my_graph(src.my_graph), label(src.label), state(undefined),
        report_changes(src.report_changes),
        led_node(src.my_graph, 1, led_body(state, label, report_changes))
    {}
    ~led() {}
    led& operator=(const led& src) {
        label = src.label; state = undefined; report_changes = src.report_changes;
        return *this;
    }
    receiver < signal_t > & get_in() { return led_node; }
    void display() {
        if (state == high) printf("%s: (*)\n", label.c_str());
        else if (state == low) printf("%s: ( )\n", label.c_str());
        else printf("%s: (u)\n", label.c_str());
    }
};

The led class contains a simple function_node that has no meaningful output (we use a continue_msg to indicate this) and thus no successors. Another way to implement this would be with an overwrite_node, but we would lose the report_changes functionality. Similarly, the digit class also cannot have successors, but we reused the gate base class to implement it, since it has multiple bits of input and needs to update its state whenever one of the inputs changes.

class digit : public gate < four_input > {
    using gate < four_input > ::my_graph;
    typedef gate < four_input > ::ports_type ports_type;
    typedef gate < four_input > ::input_port_t input_port_t;
    class digit_body {
        signal_t ports[4];
        unsigned int &state;
        string &label;
        bool& report_changes;
    public:
        digit_body(unsigned int &s, string &l, bool& r) : state(s), label(l), report_changes(r) {
            for (int i=0; i < N; ++i) ports[i] = undefined;
        }
        void operator()(const input_port_t::output_type& v, ports_type& p) {
            unsigned int new_state = 0;
            if (v.indx == 0) ports[0] = std::get < 0 > (v.result);
            else if (v.indx == 1) ports[1] = std::get < 1 > (v.result);
            else if (v.indx == 2) ports[2] = std::get < 2 > (v.result);
            else if (v.indx == 3) ports[3] = std::get < 3 > (v.result);
            if (ports[0] == high) ++new_state;
            if (ports[1] == high) new_state += 2;
            if (ports[2] == high) new_state += 4;
            if (ports[3] == high) new_state += 8;
            if (state != new_state) {
                state = new_state;
                if (report_changes) {
                    printf("%s: %x\n", label.c_str(), state);
                }
            }
        }
    };
    string label;
    unsigned int state;
    bool report_changes;
 public:
    digit(graph& g, string l, bool rc=false) :
        gate < four_input > (g, digit_body(state, label, report_changes)),
        label(l), state(0), report_changes(rc) {}
    digit(const digit& src) :
        gate < four_input > (src.my_graph, digit_body(state, label, report_changes)),
        label(src.label), state(0), report_changes(src.report_changes) {}
    ~digit() {}
    digit& operator=(const digit& src) {
        label = src.label; state = 0; report_changes = src.report_changes;
        return *this;
    }
    void display() { printf("%s: %x\n", label.c_str(), state); }
};

Because digit inherits from gate, it reuses gate’s get_in methods to connect to the ports of a digit object.

Here’s an example code to test out the four-bit adder. First, create a graph:

graph g;

Then, create the four-bit adder, some toggles with which to set the inputs to the adder, and a digit and an LED to display the output:

four_bit_adder four_adder(g);
std::vector < toggle > A(4, toggle(g));
std::vector < toggle > B(4, toggle(g));
toggle CarryIN(g);
digit Sum(g, "SUM");
led CarryOUT(g, "CarryOUT");

Next, connect our toggles to the input ports of the adder, and connect the adder’s output ports to the display devices:

for (int i=0; i<4; ++i) {
    make_edge(A[i].get_out(), four_adder.get_A(i));
    make_edge(B[i].get_out(), four_adder.get_B(i));
    make_edge(four_adder.get_out(i), Sum.get_in(i));
}
make_edge(CarryIN.get_out(), four_adder.get_CI());
make_edge(four_adder.get_CO(), CarryOUT.get_in());

Almost ready to go, activate all the switches at the low state so that everything starts at zero:

for (int i=0; i<4; ++i) {
    A[i].activate();
    B[i].activate();
}
CarryIN.activate();

Now I can start flipping toggles. I’ve set digit and led to display only when requested by default, because I don’t want to see all the changes before this circuit reaches a steady state. Let’s try 8+5:

A[3].flip();
B[0].flip();
B[2].flip();

Wait for the circuit to reach a steady state:>/p>

g.wait_for_all();

Now display the results:

Sum.display();
CarryOUT.display();

And here they are:

SUM: d
CarryOUT: ( )

And with that, I’ll wrap up this blog by saying that the logic simulation example code is available as an example in Intel® TBB 4.0 Update 4, and that it has several other interesting features, like push button and constant signal input devices, NAND and NOR gates, and a D-latch circuit example. Please let us know of other interesting use cases for the or_node and any other feedback you’d be willing to give.

To begin with, I’ll first construct a one-bit full adder as in Figure 2 below:

Figure 2

The inputs are A and B, and a Carry-in bit, and the output is the sum S, and a Carry-out bit. Here is the code for the one_bit_adder class:

class one_bit_adder {
    broadcast_node < signal_t > A_port;
    broadcast_node < signal_t > B_port;
    broadcast_node < signal_t > CI_port;
    xor_gate < two_input > FirstXOR;
    xor_gate < two_input > SecondXOR;
    and_gate < two_input > FirstAND;
    and_gate < two_input > SecondAND;
    or_gate < two_input > FirstOR;
    graph& my_graph;
    void make_connections() {
        make_edge(A_port, FirstXOR.get_in(0));
        make_edge(A_port, FirstAND.get_in(0));
        make_edge(B_port, FirstXOR.get_in(1));
        make_edge(B_port, FirstAND.get_in(1));
        make_edge(CI_port, SecondXOR.get_in(1));
        make_edge(CI_port, SecondAND.get_in(1));
        make_edge(FirstXOR.get_out(), SecondXOR.get_in(0));
        make_edge(FirstXOR.get_out(), SecondAND.get_in(0));
        make_edge(SecondAND.get_out(), FirstOR.get_in(0));
        make_edge(FirstAND.get_out(), FirstOR.get_in(1));
    }
public:
    one_bit_adder(graph& g) :
        my_graph(g), A_port(g), B_port(g), CI_port(g), FirstXOR(g),
        SecondXOR(g), FirstAND(g), SecondAND(g), FirstOR(g)
    {
        make_connections();
    }
    one_bit_adder(const one_bit_adder& src) :
        my_graph(src.my_graph), A_port(src.my_graph), B_port(src.my_graph),
        CI_port(src.my_graph), FirstXOR(src.my_graph), SecondXOR(src.my_graph),
        FirstAND(src.my_graph), SecondAND(src.my_graph), FirstOR(src.my_graph)
    {
        make_connections();
    }
    ~one_bit_adder() {}
    receiver < signal_t > & get_A() { return A_port; }
    receiver < signal_t > & get_B() { return B_port; }
    receiver < signal_t > & get_CI() { return CI_port; }
    sender < signal_t > & get_out() { return SecondXOR.get_out(); }
    sender < signal_t > & get_CO() { return FirstOR.get_out(); }
};

This implementation is almost a straightforward translation of the gates and their connections into the flow graph format. The one complication is the addition of the broadcast_nodes for each of the input ports. The reason for this is simply to enable connection to a single port from outside of the adder. Since each of the inputs is connected to two gates inside of the one_bit_adder object, there is no single port associated with them automatically. Adding the broadcast_nodes enables us to provide the methods get_A, get_B and get_CI that each return a single port capable of receiving data. So, in looking at the diagram above, you can think of the three broadcast_nodes as standing in for the black junction circles that the three inputs are connected to directly.

To make the four_bit_adder class, simply chain together a set of four one_bit_adders and connect the Carry-out port of each adder to the Carry-in port of the next adder, as shown in Figure 3 below:

Figure 3

This time, the class is even more straightforward to implement, because no broadcast_nodes are needed; every input already has exactly one internal connection.

class four_bit_adder {
    graph& my_graph;
    std::vector < one_bit_adder > four_adders;
    void make_connections() {
        make_edge(four_adders[0].get_CO(), four_adders[1].get_CI());
        make_edge(four_adders[1].get_CO(), four_adders[2].get_CI());
        make_edge(four_adders[2].get_CO(), four_adders[3].get_CI());
    }
 public:
    four_bit_adder(graph& g) : my_graph(g), four_adders(4, one_bit_adder(g)) {
        make_connections();
    }
    four_bit_adder(const four_bit_adder& src) :
        my_graph(src.my_graph), four_adders(4, one_bit_adder(src.my_graph))
    {
        make_connections();
    }
    ~four_bit_adder() {}
    receiver < signal_t > & get_A(size_t bit) {
        return four_adders[bit].get_A();
    }
    receiver < signal_t > & get_B(size_t bit) {
        return four_adders[bit].get_B();
    }
    receiver < signal_t > & get_CI() {
        return four_adders[0].get_CI();
    }
    sender < signal_t > & get_out(size_t bit) {
        return four_adders[bit].get_out();
    }
    sender < signal_t > & get_CO() {
        return four_adders[3].get_CO();
    }
};

Here, the constructor makes a vector of exactly four adders, and connects the Carry-out ports to the Carry-in ports as appropriate. The multi-bit inputs and outputs have port access methods that take a bit as a parameter. So for example, to get the input port for bit 2 of input B, you would use get_B(2).

In Part 3, I will present some interesting input and output devices to add to the logic simulation library, and with those, I’ll put together a small simulation that shows the four_bit_adder in action.

Consider an AND gate. In its simplest form, it takes two inputs, and produces a single output. The first thing that comes to mind to represent this is the flow graph function_node: it could take a pair as input, and a body that computes the logical AND operation on the items in the pair, and puts out the result as its output. That might work, but let’s think a little more about how such a gate might receive its two input signals: a function_node takes a single argument, so I’d have to group the two inputs together. However, both inputs will be coming from different senders, and may not be available at the same time. Should I preface the function_node with a join_node? Possibly, but there’s still a limitation with a join_node: it gathers together the inputs and when it has received the full complement, it then sends them along as a tuple. But this still isn’t exactly the behavior I want. What I really want is when either of the inputs becomes available, the function_node should be told about it, because it will need to change its output value when any of its input values change.

Thus, the first decision about gates is this: Gates are responsive: when any input changes, the gate will check if its output needs to change. To simplify this a little, and make our flow graph have to do a little less work, I’ll make this second decision: Gates are lazy; a gate will send data to its output port only when that data differs from the previous value sent to that output port. This will certainly reduce the number of tasks doing redundant work in the graph.

So, on the input side, something reports changes on any input port, and on the output side, something produces output, or not, depending on if the output value has changed. Neither of these behaviors corresponds exactly to a function_node. However, the new feature multifunction_node (formerly the Community Preview feature (CPF) multioutput_function_node) can certainly meet the output needs: it can optionally produce an output. For the input, if the title of this blog hasn’t given it away already, my choice is the or_node. The or_node will pass along any input it receives on any input port at any time, giving exactly the responsiveness I need. The or_node is currently a CPF in Intel® TBB.

Figure 1

Figure 1 illustrates this basic logic gate design. Note that the or_node takes a variable number of inputs – no need to limit it to two – and the multifunction_node takes a body that produces either no output or one output. In general, the multifunction_node can produce zero or more outputs of varying types, but for the gate implementation, zero or one output will suffice. Let’s take a look at the actual code for this, the gate template class.

First, I set up a type signal_t to represent the signal data being transferred. Since I’m allowing the gates to fire only when the output state changes, it helps to have an additional undefined state for initialization.

typedef enum { low=0, high, undefined } signal_t;

Next, I define a few potential input configurations to gates. I could go all out and add eight_input gates, but I couldn’t dredge up a use for them from the dark and rarely-visited corner of my brain where I keep the knowledge leftover from a digital logic course so many years ago.

typedef tuple < signal_t > one_input;
typedef tuple < signal_t, signal_t > two_input;
typedef tuple < signal_t, signal_t, signal_t > three_input;
typedef tuple < signal_t, signal_t, signal_t, signal_t > four_input;

Now I’m ready to set up the gate template.

template < typename GateInput >
class gate {
protected:
    typedef or_node < GateInput > input_port_t;
    typedef multifunction_node < typename input_port_t::output_type, tuple < signal_t > > gate_fn_t;
    typedef typename gate_fn_t::output_ports_type ports_type;
public:
    static const int N = std::tuple_size < GateInput > ::value;

    template < typename Body >
    gate(graph& g, Body b) : my_graph(g), in_ports(g), gate_fn(g, 1, b) {
        make_edge(in_ports, gate_fn);
    }
    virtual ~gate() {}
    virtual gate& operator=(const gate& src) { return *this; }
    sender < signal_t > & get_out() { return output_port < 0 > (gate_fn); }
    receiver < signal_t > & get_in(size_t port) {
        return gate_helper < N > ::get_inport(in_ports, (int)port);
    }
protected:
    graph& my_graph;
private:
    input_port_t in_ports;
    gate_fn_t gate_fn;
};

The class is templated by the input configuration, GateInput, so for example, I would pass in two_input if I wanted to make a gate with two inputs. Then I define two types. First, input_port_t, which is the type of the or_node that that I’ll pass the input configuration to, as specified by GateInput. Second is gate_fn_t, which is the multifunction_node that takes the output from the or_node, performs the function of the gate, and outputs a single signal_t (or nothing). These types are used to declare the actual graph nodes in_ports and gate_fn, in the private section of the class above.

The gate constructor initializes the two graph nodes, making them belong to a graph g that is passed in as a reference parameter. Additionally, the constructor takes a function object b that performs the actual logical operation on the inputs to the gate, and determines what the new output will be. So in the case of an AND gate, I would pass in a function object that computes a logical AND operation. The constructor also completes this small component by connecting the two graph nodes with the make_edge function.

In order to connect this gate to other components, I’ve provided methods to access the input ports and the output port. get_in takes a port number and returns a reference to an input port capable of receiving data, i.e. a receiver& in the flow graph jargon. It uses the gate_helper::get_inport function shown below to extract the input port to the or_node. get_out returns a reference to the output port of the multifunction_node which is capable of sending data, i.e. a sender&.

template < int N >
struct gate_helper {
    template < typename TupleType >
    static inline receiver < signal_t > & get_inport(or_node < TupleType > & in_ports, int port) {
        if (N-1 == port) return input_port < N-1 > (in_ports);
        else return gate_helper < N-1 > ::get_inport(in_ports, port);
    }
};
template < >
struct gate_helper < 1 > {
    template < typename TupleType >
    static inline receiver < signal_t > & get_inport(or_node < TupleType > & in_ports, int port) {
        return input_port < 0 > (in_ports);
    }
};

Now that I have a building block for creating a wide variety of logic gates, I’ll use it for designing an AND gate. When creating the derived class and_gate, the main purpose is to define the functor that gets passed to the gate_fn object inside the gate base class. and_body computes a logical AND operation over all the inputs to the gate, including undefined inputs, so the function is not completely trivial.

template < typename GateInput >
class and_gate : public gate < GateInput > {
    using gate < GateInput > ::N;
    using gate < GateInput > ::my_graph;
    typedef typename gate < GateInput > ::ports_type ports_type;
    typedef typename gate < GateInput > ::input_port_t input_port_t;
    class and_body {
        signal_t ports[N];
        signal_t state;
        bool touched;
    public:
        and_body() : state(undefined), touched(false)
            for (int i=0; i < N; ++i) ports[i] = undefined;
        }
        void operator()(const typename input_port_t::output_type& v, ports_type& p) {
            ports[v.indx] = or_output_helper < N > ::get_or_output(v);
            signal_t new_state=high;
            size_t i=0;
            while (i < N) {
                if (ports[i] == low)
                    new_state = low; break;
                else if (ports[i] == undefined && new_state != low)
                    new_state = undefined;
                ++i;
            }
            if (!touched || state != new_state) {
                state = new_state;
                std::get < 0 > (p).try_put(state);
                touched = true;
            }
        }
    };
 public:
    and_gate(graph& g) : gate < GateInput > (g, and_body()) {}
    and_gate(const and_gate < GateInput > & src) : gate < GateInput > (src.my_graph, and_body()) {}
    ~and_gate() {}
};

The and_body keeps track of the states of the gate’s input ports and output port. These are all initially undefined. The operator() for and_body receives the or_node output in parameter v, which indicates that data was received on one of the input ports. The input port that received data is specified in v.indx. Accessing the data from that port is a little more challenging, as the entire input tuple is passed in v.result. I wrote a helper function or_output_helper::get_or_output to select the v.indx-th port of the tuple v.result. This value is used to update the locally stored state of the appropriate port, and then the new output state is calculated. The new state is checked to see if it differs from the old state, and if so, the new state is sent out on the appropriate output port of the multifunction_node (which in this case, since there is only one output, is always port zero). Note also that the very first time a gate receives data, i.e. when touched is false, the new state is sent out even if it is not different from the initial state. This is useful when the gate is a part of a larger circuit. It allows any initial settings on input ports to propagate through the graph and register at any possible output devices that might exist.

The helper function that extracts the or_node output is as follows:

template < int N >
struct or_output_helper {
    template < typename OrOutputType >
    static inline signal_t get_or_output(const OrOutputType& out) {
        if (N-1 == out.indx) return std::get < N-1 > (out.result);
        else return or_output_helper < N-1 > ::get_or_output(out);
    }
};
template < >
struct or_output_helper < 1 > {
    template < typename OrOutputType >
    static inline signal_t get_or_output(const OrOutputType& out) {
        return std::get < 0 > (out.result);
    }
};

Given an AND gate, it’s easy to see how to make OR gates and any other sort of basic logic gate from the base class gate.

As the or_node is currently a Community Preview feature, it’s a good time to have a look at it and give us your feedback.

In Part 2 of this blog, I’ll show you how to put together a variety of basic logic gates to make a four-bit adder.

There are two modes of use for this feature: basic mode and expert mode. Basic mode is straightforward and not much more complex than using a mutex. Expert mode requires some understanding of how the aggregator works, and additional coding, but can enable additional performance improvements. In this blog, I will first illustrate how to use the aggregator in the basic mode. Then I’ll give a brief overview of how the aggregator works, followed by an example of how to use the aggregator in the expert mode. Finally, I’ll examine the performance of the aggregator and suggest approaches to help decide whether or not to use it.

Side-by-side Comparison of Basic Aggregator Usage with Mutex Usage

In this simple example, I’ll compare the usage of a mutex with an aggregator to lock push and pop operations on a serial priority queue object of type std::priority_queue. This example uses C++1x features, such as lambdas, but one could use function objects instead. Fair warning: I’m interspersing code snippets below, because this blog format doesn’t allow for side-by-side code comparison. Please don’t try to use both a mutex and an aggregator to protect the same code.

First, declare the priority queue. I'll use a simple integer priority queue here:

typedef int value_type;
typedef priority_queue < value_type, std::vector < value_type > , compare_type > pq_t;
pq_t my_pq;

Declare a mutex to protect my_pq:

spin_mutex my_mutex;

Alternatively, declare an aggregator to protect my_pq:

aggregator my_aggregator;

Declare an element to push/pop from queue:

value_type elem = 42;

Now, push an element on the queue using the mutex:

{
    tbb::spin_mutex::scoped_lock my_lock(my_mutex);
    my_pq.push(elem);
}

Or, push the element on the queue using the aggregator and a lambda expression:

my_aggregator.execute( [&my_pq, &elem](){
    my_pq.push(elem);
} );

Pop an element off the queue using the mutex:

bool result = false;
{
    tbb::spin_mutex::scoped_lock my_lock(*my_mutex);
    if (!my_pq.empty()) {
        result = true;
        elem = my_pq.top();
        my_pq.pop();
    }
}

Pop an element off the queue using the aggregator:

bool result = false;
my_aggregator.execute( [&my_pq, &elem, &result](){
    if (!my_pq.empty()) {
        result = true;
        elem = my_pq.top();
        my_pq.pop();
    }
} );

How the Aggregator Works

As we see above, the usage of the aggregator in basic mode is trivially different from using a mutex. However, it is clearly working in a different way. In order to execute a critical section, you pass it to an aggregator via the execute method. When the execute method returns, the critical section has been executed, but how this happened is hidden inside the black box of the aggregator.

Looking at the header file aggregator.h that defines the aggregator, these details become clear. To use the aggregator in expert mode, you should have some familiarity with the header file, and I'll guide you through the most important features in the rest of this blog.

First note that aggregator inherits from a class aggregator_ext that takes a template parameter. Aggregator instantiates that template parameter with a simple handler defined in the header, handler_type = internal::basic_handler. We will discuss this more later.

The execute method of aggregator takes a function body as parameter, and encapsulates body in a basic_operation object, which inherits from aggregator_operation. Aggregator_operations are sent to the aggregator_ext’s mailbox where they may concurrently accumulate while they await execution. One thread, the active handler, i.e. the first thread to place an aggregator_operation in the empty mailbox, will grab all the operations that have accumulated there, effectively emptying the mailbox. It will then go through all the operations that it grabbed, and serially execute the function bodies stored in those objects. The mechanism used to execute function bodies is specified by aggregator_ext’s template parameter, which in the default case is called basic_handler.

This basic_handler is straightforward in its functioning: it is passed the list of aggregator_operations, and it loops through this list and handles each item. It makes use of a few methods on aggregator_operation to do this properly: next is used to traverse to the next operation in the list, start prepares the operation to be handled, and finish is called after the operation is handled to inform the thread waiting on the execution of the operation that the operation is completed. When all operations are handled, the active handler thread can leave the aggregator, since its own call to execute has been satisfied in the process.

The details of the synchronization that make this all possible can be found in aggregator.h. We won’t explain them fully here, because we already have enough information to proceed to use the aggregator in expert mode. It is enough to know that threads hand over critical sections to the aggregator, and one of these threads will execute all the operations serially on behalf of the other threads as a single critical section.

Using the Aggregator in Expert Mode

I’ll use the same example as before, allowing threads to safely push and pop to a serial std::priority_queue. The expert mode of aggregator allows the user to pass any sort of data in to the aggregator as an aggregator_operation via the process method (note the different method name – we were using execute in basic mode), along with an aggregating function object that is called by the active handler to perform the serial execution of operations. In this case, I’ll pass data about a push or pop operation to the aggregator via process, and provide a custom function object to perform the operations.

First, create a class derived from aggregator_operation to hold the operation data.

class op_data : public aggregator_operation {
public:
    value_type* elem;
    bool success;
    bool is_push;
    op_data(value_type* e, bool push=false): elem(e), success(false), is_push(push) {}
};

Then, create a handler to pass in as the aggregator’s template parameter:

class my_handler_t {
    pq_t *pq;
public:
    my_handler_t() {}
    my_handler_t(pq_t *pq_) : pq(pq_) {}
    void operator()(aggregator_node* op_list) {
        op_data* tmp;
        while (op_list) {
            tmp = (op_data*)op_list;
            op_list = op_list->next();
            tmp->start();
            // handle tmp here
            if (tmp->is_push) pq->push(*(tmp->elem));
            else {
                if (!pq->empty()) {
                    tmp->success = true;
                    *(tmp->elem) = pq->top();
                    pq->pop();
                }
            }
            // done handling tmp
            tmp->finish();
        }
    }
};

Now, to create an aggregator, use the aggregator_ext type name and pass this handler’s type in as the template parameter, and initialize the handler and pass it in as an argument to the constructor:

aggregator_ext < my_handler_t > my_aggregator(my_handler_t(my_pq));

To perform a push, simply create the op_data node with the push information and pass it to process:

op_data my_push_op(&elem, true);
my_aggregator.process(&my_push_op);

And to perform a pop:

bool result;
op_data my_pop_op(&elem);
my_aggregator.process(&my_pop_op);
result = my_pop_op.success;

When to use Aggregator and why use Expert Mode?

A good way to start is to compare the performance of your code using your current locking mechanism to a version of your code that uses an aggregator instead. In practice, we (developers of TBB) have often found that a mutex is sufficient and outperforms aggregator when contention on the critical region is low. For higher contention, we often find that the use of the aggregator is justified.

The aggregator provides most of its performance improvements in hot cache execution of operations on a single thread. (Recall the active handler?) Thus, the more concurrent contention on your critical region, the larger the aggregations will be that are assembled, and the greater the benefits of executing operations with a hot cache on a single thread.

If you do find that the basic aggregator improves your code’s performance, consider moving to the expert level. To begin with, you can simply transform your code as I’ve shown in the expert example above. This should result in better performance over the basic interface. The reason for this is that, in the basic interface, the function object or lambda expression you wish to execute and all the references to data that you want that code to access are stored on the stack of the thread that originated the operation. Referring back to the basic example, this means that for each operation, we look up a different reference to the same priority queue. But, in the expert example above, note that we store just a few data references in the aggregator_operation, and the code to execute the operation and references to the shared data (my_pq) are local to the aggregating functor and only need to be looked up once to handle all the operations in an aggregation. This enhances the hot cache effect by reducing the quantity of non-local stack accesses.

The expert-level usage of aggregator shown above is quite straightforward. However, you are free to handle operations in the aggregating handler in whatever manner you like. Consider the aggregation of operations an opportunity to develop new and interesting serial algorithms. This gives you a unique opportunity to make use of a kind of lookahead capability: you know the set of operations that you need to perform. For example, Intel® TBB’s concurrent_priority_queue handles the operations in two passes, performing some of them and postponing others, because some orderings of operations are more efficient than others. The only rules for processing operations in the aggregating handler are that they should all be handled, and, in some cases, there should be some serial sequence of the operations that achieves the same result (i.e. sequential consistency).

I’d like to hear about your experiences using aggregator, so if you get a chance, give it a try, and let me know how it went! You can comment here, or better yet, start a discussion on the Intel® TBB forum.

Motivation

We had vague requests from customers to implement a memory pool (Wikipedia calls it region) or some of its properties in the TBB scalable memory allocator. We summarized these requests and general information on memory pools from the Internet and got the following compilation of major properties and abilities:

• Memory pools basically do the same job as standard memory allocators but additionally group memory objects under umbrella of a specific pool instance which enables:
  • fast deallocation of all the memory at once on pool destruction or for sake of further reuse
  • less memory fragmentation and related synchronization between independent groups
• Memory pools allow more control over acquisition and release of memory resources, and may have user-specific sources of memory:
  • memory chunk/buffer of a fixed size
  • redirection to a specific memory provider, e.g. standard or custom implementation of malloc, big memory pages, memory tied to specific NUMA node, IPC shmem regions.

To squeeze more performance and to fight memory fragmentation, some specific implementations allocate objects of fixed size only (so called object pools, e.g. boost::pool, Wikipedia calls it memory pool) or are unable to deallocate individual object ("arena allocator"). In our implementation, we tried to provide more general functionality in thread-safe and scalable way. For that purpose, the implementation of the memory pools is based on TBB scalable memory allocator and so has similar speed and memory consumption properties. Later we may address more specific use cases, based on the feedback.

Usage

Our memory pools API consists of two classes for thread-safe memory management: tbb::fixed_pool and tbb::memory_pool. The first one is for the simple case when an already allocated memory block and is used for allocation of smaller objects. And the second one utilizes a user-specified memory provider to obtain big chunks of memory where smaller objects reside. As opposed to fixed_pool, memory_pool is able to grow on demand and relinquish unused chunks back to the provider.

Both classes provide familiar methods for allocation and deallocation:

void *ptr = my_pool.malloc( (size_t) 10 );  // allocate 10 bytes
ptr = my_pool.realloc( ptr, (size_t) 12 );  // extend the allocation to 12 bytes
my_pool.free( ptr );                        // deallocate it

Additionally, there is a method which deallocates all the memory at once, i.e. it is a faster equivalent to a series of calls to my_pool.free() for each pointer obtained in this pool by previous calls to my_pool.malloc():

my_pool.recycle();  // Frees all the memory in the pool for reuse

Please note, that it is not thread-safe to call it concurrently to other methods on the same instance (similarly to clear() method in containers).
We also provide an (almost, except absence of default constructor) STL-compliant allocator class to enable pools inside STL containers:

typedef tbb::memory_pool_allocator<int> pool_allocator_t;
std::list<int, pool_allocator_t> my_list( (pool_allocator_t( my_pool )) );

Now, the only thing that holds us back from the first experiment with this new feature of TBB is the question – how to create the ‘my_pool’.  First, we need to enable this feature and include the header:

#define TBB_PREVIEW_MEMORY_POOL 1
#include “tbb/memory_pool.h”

If you want to create a memory pool on top of your memory block, let’s specify its address and size in bytes to the constructor of tbb::fixed_pool class, as in following excerpt:

char buffer[1024*1024];
// The casts below are just to show the types of arguments.
tbb::fixed_pool my_pool( (void*)buffer, (size_t)1024*1024*sizeof(char) );

The maximal amount of memory which can be allocated from the pool declared above is limited by size of the buffer minus some space for control structures. And if you want to avoid this limitation, let’s use tbb::memory_pool template class specifying memory provider (which will be discussed later) as its template argument:

tbb::memory_pool< std::allocator<char> > my_pool(/*optionally: allocator instance*/);

You can specify any STL-compatible allocator as the memory provider (though this is a subject to change). It will provide (big) memory chunks for  my_pool when necessary. The destructor of the memory_pool class implies release of all the memory chunks back to the memory provider.

Let’s consolidate our knowledge in one artificial example:

// Link this with tbbmalloc library
#define TBB_PREVIEW_MEMORY_POOL 1
#include "tbb/memory_pool.h"
#include <list>
#include <stdio.h>

void main() {
    static char buf[1024*1024*4]; // buffer for interim data
    tbb::fixed_pool interim_pool(buf, sizeof(buf)); // pool for temporary objects
    tbb::memory_pool< std::allocator<char> > result_pool; // pool to store the results

    typedef tbb::memory_pool_allocator<int> result_allocator_t; // interface to STL containers
    std::list<int, result_allocator_t> result_list( (result_allocator_t( result_pool )) );

    for(int result = 0, i = 0; i < 100; i++, result = 0) {
        for(int j = 0; j < 1000000; j++) {
            int *p = (int*)interim_pool.malloc(4);
            if( p ) result++; // really dummy :)
        }
        // in real application, here can be some processing of allocated objects
        result_list.push_back(result); // no memory fragmentation here - separate pool
        interim_pool.recycle(); // free all the interim objects
        printf("%d\n", result); // should be the same number on each iteration
    }
} // all the memory is released back implicitly

The simple part is done, and I hope that you are interested enough to proceed with more complex questions, and tell us what you think about it.

Someone may want to know whether it is possible to construct a pool in a memory allocated form another pool. It is possible, but one should take care to destroy the inner pool prior to destruction of the outer pool or a call to recycle(). Do you know a good reason to enable such a nesting?

Memory provider interface

From an API designer perspective, the memory provider is the most questionable part of the scalable pools API. And since it is yet a community preview feature, you are welcome to influence its design. Curious readers might want to ask questions like the following:

• what are the requirements for the template argument?
• why is std::allocator used as a memory provider?
• why the type used with std::allocator in examples above is “char”?

The template argument of tbb::memory_pool accepts a memory provider class which satisfies minimal requirements of STL compatible allocator according to the last C++11 standard: allocate and deallocate methods, and a value_type definition.

Using std::allocator and compatible classes is perhaps the most straight-forward way to enable memory_pool anywhere. However from efficiency standpoint, it makes probably not much sense because such allocators are intended for rather small objects by design while memory provider should operate with megabytes. For users who don’t care what the memory provider is, we could better provide a default one instead which would map to system-default way for memory mapping.

And finally, TBB memory pools don’t really need the type of allocation (i.e. char in the declaration of tbb::memory_pool<std::allocator<char>>), but rather need to know the granularity of requests to the memory provider. And this is not only specification for type of arguments for allocate and deallocate, this information is used in our implementation to determine size of memory requests to memory provider. For example, consider big pages which can be mapped only by chunks of megabytes:

// A custom memory provider for memory_pool
class big_pages {
public:
    typedef char[2*1024*1024] value_type;
    void *allocate(size_t pages) {
        return mmap(0x0UL, pages*2*1024*1024, PROT_READ | PROT_WRITE,
                    MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, 0, 0);
    }
    // the pointer type requirement is also actually relaxed
    void deallocate(void *ptr, size_t pages) {
        munmap(ptr, pages*2*1024*1024);
    }
};
// usage:
tbb::memory_pool<big_pages> my_pool;

Some food for thoughts

The way granularity is specified in the line 4 in the above example is not straight-forward and can be viewed as confusing. This is the price of STL-compliant interface of the memory provider and we are not sure if it has more pros than cons:

• STL compatibility is supposed to reuse widely implemented memory allocators.
  • On the other hand, these allocators are usually purposed for small sizes of allocations but a pool will need memory chunks of at least hundreds of kilobytes.
• In theory, it allows easy nesting of memory pools using our memory_pool_allocator class.
  • But we studied that nesting of the pool in some other implementations does not mean reusing the memory allocated by parent pool but rather a hierarchy of pool objects.
  • And such a nesting is not yet supported anyway
• It is easier to remember the requirements based on well-known standard interface
• Granularity is a property of the memory provider and must be passed along with it

As an alternative interface, we consider to make the granularity explicitly specified but in a separate trait class which should be specialized only for the memory providers with granularity of allocations > 1. It is even possible to keep STL-compatibility using metaprogramming magic, e.g. define the granularity to sizeof(value_type) if value_type defined.

Another question is how to introduce alignment in the interface of memory pools. Basically, it can be either aligned_malloc() and aligned_realloc(), or an optional argument for malloc() and realloc() methods.

Also, are the suggested class names good, or do we need to find better names (for instance, "fixed_region" and "dynamic_region" to align with terms of Wikipedia)?

Feedback is very welcome

We are very eager to hear from you what do you think about above and how can it be used in your projects.

5. Intel® Cilk Plus open source port to GCC - Intel® Cilk Plus was announced in 2010, and an open source specification has been out since late 2010 as well. However this year we began, along with the open source community, to port Cilk Plus to GCC. Some of the first items ported were the parallelism keywords, which is significant to me because it makes our Cilk Plus parallelism model available to a greater audience.

4. Intel® VTune™ Amplifier XE and Intel® Inspector XE MPI Support - In the new Cluster Studio XE product, VTune Amplifier XE and Inspector XE are now MPI-enabled. This is important because we are beginning to see more hybrid programming in the HPC and cluster world - which means the applications use a combination of MPI and another threading model (such as OpenMP, Cilk Plus, or Intel® Threading Building Blocks). We have an existing product, Intel® Trace Analyzer and Collector, that analyzes MPI efficiency for a cluster app, but analyzing performance of an individual process running on an MPI rank was more difficult. Now we make it easier to use VTune Amplifier XE or Inspector XE to analyze the threading model used within a rank, which helps us support more cluster customers.

3. Intel® Threading Building Blocks Flow Graph - I was introduced to flow graph this year, when I worked with my colleague Victoria Gromova to create some TBB labs for Intel Developer Forum. Victoria wanted to highlight flow graph as one of the new features of TBB 4.0. Flow graph is a new construct that supports many more types of control algorithms, like dependency graphs, event-based models or reactive-based flows. In short, it opens up TBB to more customers while maintaining or improving the TBB performance we have come to expect.

2. VTune Amplifier XE attach to running process on Linux* - This is a great example of our development team responding to customer feedback. Being able to analyze a running process for a defined period of time (instead of launching it) has been requested by many of our clients. We first got this implemented on Windows*, then this September provided the feature for Linux* in Intel® Parallel Studio XE Service Pack 1. I have already been visiting some users who requested this and it is great to be able to share that the feature they have been asking for is here!

1. VTune Amplifier XE interface for Intel® Microarchitecture Codename Sandy Bridge - For readers of my blog this one should not be a surprise! I have created quite a bit of training material on these new Sandy Bridge features. We now provide an analysis type for Sandy Bridge that helps users easily identify the most common software performance issues at the microarchitectural level, and it includes pre-coded metrics, thresholds, and issue highlighting for usability. This is my favorite new feature because, even though I am not a developer, I got to help a little with making this interface by helping define some performance metrics and thresholds and validating them on workloads. It is very cool to see my contributions in the product.

There you have it! I hope you have a chance to try out some of our new product features now or in the coming year. Let us know your favorites, or your requests.

At LinuxConf (LCA2010) James Reinders gave a talk about the Threading Building Blocks (TBB) library, a C++ threading library that sets out to make multicore programming more accessible to the average programmer. We took this idea on board and explored the possibilities of opening up this approach to an even wider audience, namely the audience of web application developers working in script languages.

Many websites require a non-trivial amount of per-request processing in the application layer, perhaps to retrieve, consolidate or otherwise manipulate data. Achieving better performance at this level improves response times and the overall user experience. Even when processing time at application level is not critical, parallelizing access to database and web service back-end layers can yield substantial improvements in perceived performance.

This drove our goal of adding TBB support into PHP and Perl, starting with HipHop as the PHP implementation of choice and later on adding Perl support to the game.

HipHop is a PHP to C++ cross compiler that was developed by Facebook to cut down on resource needs and speed up the execution times of their gigantic web infrastructure that was started on a classic PHP/MySQL stack and now has to scale to hundreds of millions of users. The HipHop project is a PHP implementation that is thread safe and already uses TBB for some memory management. We started by extending the existing support and added first only the new *parallel_for* function. Later, we added concurrent data structures and re-implemented our first approach.

What we have now is a robust implementation of *parallel_for* and *parallel_reduce* with the data structures needed to support them. What we learned on the way was both, very enlightening and quite frustrating at times. Our aim to make TBB more widely accessible was reached by getting the language extension into HipHop but we also tried to get it into Zend PHP. This turned out to only work with a language compatibility module that does not provide the full glory we can offer on the HipHop platform. The reason for this is the architecture of the PHP interpreter.

Implementing threading into language interpreters turns out to be very hard. There are two dormant/failed approaches in Perl and every attempt in PHP has failed so far. The core developers on both sides are very much in doubt if it is a path worth going down at all. The problem is global locking and copying/sharing of data structures that are thread local. Our Perl implementation is a starting point that could influence not only the Perl community but other interpreter designers and interpreter developers as well.

In the Perl community we are trying to lobby for a const keyword that would lock a data structure and remove the need to copy it into every thread. The ability to make something immutable is missing in Perl and PHP and this makes the startup cost of any worker thread very expensive. For the Perl library we wrote a lazy clone module that would only clone a data structure if the worker thread really accesses it. That way we only penalize the worker thread for accessing data - we can possibly get around cloning structures at all if they are not accessed within this task.

In our work with the PHP HipHop compiler we also wrote a patch set for WordPress and enhanced WordPress with our new *parallel_for* language extension. This trial brought us instant success in reduced page load times. The patch set for WordPress only replaced some key *foreach* loops with *parallel_for* and was our first real success with the TBB library in PHP. Based on that success we started out to re-implement our initial approach and tidy up our patch set for HipHop to make it more accessible to others.

The Perl project worked towards a Perl module that can be used to get access to TBB functions directly. We also started out to implement the core memory structures and then built on top of those the *parallel_for* functionality. The module we have now is stable enough to demonstrate the gains we can get by using TBB in Perl.

To round the project off we implemented two little tools as real world demo and as working code to look at. The demo is based around the HTML5 geo tag which is present in modern browsers and can be read with a Javascript API. In the HipHop version we use it to read the current Lat/Lon from the accessing browser and then parse the Twitter firehose to find tweets with embedded image URLs.

In the Perl demo we query Flickr and fetch a grid of 4x4 images, cache them locally and then render one big image out of scaled versions of the single images. The demos are running on geopic.me

To sum up our experience with TBB and script languages we know now that threading interpreters buries its very own set of challenges but we were able to get further than others did on the same mission by using TBB. The libraries we produced so far - which are open source and can be found on our github account - will be further developed and maintained.

We will continue working on both platforms to expose the power of multicore CPUs to developers in an approachable way. Along the way we also produced a number of more detailed white papers covering various aspects of the project:

* threads::tbb
* TBB in WordPress
* WordPress on HipHop

Get in touch if you are interested in these projects or have questions about the work we did. There is further information on our website OpenParallel.com

Contact: Nicolas Erdody

A question was recently submitted about an implementation of a pipeline using a flow graph.  That question made me realize that pointing out some of the subtle differences between an Intel TBB pipeline and an Intel TBB flow graph might prevent some confusion among users.

As a running example, I will convert the pipeline example that ships with the Intel TBB distribution, examples/pipeline/square. This example uses the three-stage pipeline shown in Figure 1.

Figure 1: The three-stage pipeline using in examples/pipeline/square.

In the square example, the input filter reads blocks of strings from a input file.  Each block is of a fixed size and when a complete block is read, it is sent to the next filter transform. The second filter is a parallel filter that allocates an output buffer, converts each string in the input buffer into a long, squares the value, and then writes the result to the output buffer. It passes the output buffer to the final filter, output, which is a serial-in-order filter. The output filter processes buffers one at a time in the order that they were created by the input filter. It dumps the resulting squared numbers to an output file. Because the buffers are processed in-order, the values written to the output file match the ordering of their corresponding values in the input file.

Figure 2 shows a straightforward conversion of this example to a flow graph. The input filter becomes a source_node, the second filter becomes a function_node with unlimited concurrency, and the final filter becomes a function_node with a concurrency of 1.   For the most part, the rectangles have become circles :)   But unfortunately, this implementation will both generate incorrect results and perform poorly.

Figure 2: A straightforward (but incorrect) translation to a flow graph.

First, let's address the correctness issue. The final node in our flow graph has a concurrency of 1, and so is serial; however it operates on the buffers in first-in-first-out order. The pipeline version, on the other hand, used a serial-in-order-filter, which allows it to re-establish the order in which the items were generated by the input filter. We can mimic this behavior by adding a sequencer node, as shown in Figure 3.

Figure 3: A correct (but slow) translation to a flow graph.

Adding a sequencer_node to our flow graph requires that we assign a sequence number to each buffer as it is allocated at the input filter. We also then must write a function that can return that sequence value for given a buffer, and provide this function to the sequencer_node.

The flow graph shown in Figure 3 still has a problem. The source_node input is directly attached to a function_node transform that has unlimited concurrency. This means that transform will accept everything that input sends to it. The input node will happily keep allocating new buffers, flooding the flow graph with inputs. In the best case, this will bog down the system resulting in poor performance. In the worst case, this could cause the application to run out of memory and crash.

Such a situation is not possible in an Intel TBB pipeline because of its token-based scheduling mechanism. When a pipeline is run, the user provides a fixed number of tokens to the pipeline. There will be at most one item per token in the pipeline. In the square example, the pipeline is run with 4 tokens per core:

pipeline.run( nthreads*4 );

A flow graph does not have token-based scheduling. Because nodes in a flow graph can generate multiple outputs, join messages together and split messages apart, it becomes difficult if not impossible to implement a token-based system that does not provide the potential for deadlock. However, the flow graph does provide alternative methods for restricting memory use. One of these methods is the use of a limiter_node, as shown in Figure 4.

Figure 4: A correct and efficient translation to a flow graph.

In Figure 4, a limiter_node is inserted between the input filter and the transform filter. A limiter_node has a user-set threshold of messages that it will allow to pass through before rejecting additional messages. It also has a second input port that can be used to decrement its internal count of messages it has forwarded. If the limiter_node in Figure 4 is constructed with a threshold of 4*nthreads it will cap memory use similarly to the pipeline implementation. The edge from the output filter back to the limiter_node will signal the node to allow additional inputs in from the input filter.

Although it is not demonstrated in the square example, a pipeline can also have a parallel input filter. In contrast, a source_node is always serial. This restriction is again due to the lack of token-based scheduling in the flow graph. The number of active copies of a pipeline’s input filter is bounded by the number of available tokens. In a flow graph there would be no such bound, and no practical way to determine an appropriate number of source_node bodies to activate concurrently. There are ways to mimic the behavior of a parallel input filter in a flow graph, but I’ll leave that discussion for a future blog post.

Below is the snippet of code from examples/pipeline/square that sets up the structure of the Intel TBB pipeline shown in Figure 1. Each of the three filters is created and then added to the pipeline. Finally the pipeline is run with 4*nthreads tokens.

tbb::pipeline pipeline;

MyInputFilter input_filter( input_file );
pipeline.add_filter( input_filter );

MyTransformFilter transform_filter;
pipeline.add_filter( transform_filter );

MyOutputFilter output_filter( output_file );
pipeline.add_filter( output_filter );

pipeline.run( nthreads*4 );

Code that can be used to set up the structure of the Intel TBB flow graph shown in Figure 4 is shown below. Each of the nodes is created and then edges are added between them. Note that the limiter_node is passed, 4*nthreads as its threshold, and that a sequencer_node is placed before output. Finally, the source_node is activated and then wait_for_all is called on the flow graph to wait for it to complete.

tbb::flow::graph g;

tbb::flow::limiter_node limiter( g, nthreads*4 );
tbb::flow::sequencer_node< TextSlice * > sequencer(g, sequencer_body() );

tbb::flow::source_node input( g, MyInputFilter(input_file), false );
tbb::flow::function_node transform( g, tbb::flow::unlimited, MyTransformFilter() );
tbb::flow::function_node output( g, tbb::flow::serial, MyOutputFilter( output_file ) );

tbb::flow::make_edge( input, limiter );
tbb::flow::make_edge( limiter, transform );
tbb::flow::make_edge( transform, sequencer );
tbb::flow::make_edge( sequencer, output );
tbb::flow::make_edge( output, limiter.decrement );

input.activate();
g.wait_for_all();

The flow graph code is clearly more verbose than the pipeline version, but their performance will be comparable. As described here, applications that are structured like pipelines will be more easily expressed using the Intel TBB pipeline, but the Intel TBB flow graph has a more flexible API and therefore can express applications that are not amenable to a linear pipeline.

In conclusion, there are subtle but important differences between an Intel TBB pipeline and an Intel TBB flow graph. First, pipeline supports serial-in-order nodes, while the flow graph requires a sequencer_node to get similar behavior.  Second, and more importantly, a pipeline uses token-based scheduling while a flow graph does not. The flow graph API does include alternatives to token-based scheduling such as the limiter_node discussed in this post.  In an earlier post,
A feature-detection example using the Intel® Threading Building Blocks flow graph, I demonstrated another alternative for managing memory use, a reserving join node. And finally, because of the lack of token-based scheduling, a source_node is always serial and more complex methods must be used to mimic a pipeline’s parallel input filter.

To learn more about other features in Intel® Threading Building Blocks 4.0, visit http://www.threadingbuildingblocks.org or to learn more about the Intel® TBB flow graph, check-out the other blog articles at http://software.intel.com/en-us/blogs/tag/flow_graph/.

One of the new Community Preview nodes is the multioutput_function_node. This node may be connected to one or more (currently up to 10) ports. The ports are passed as a std::tuple to each execution of the node’s functor, and during each execution the functor may send items to any combination of output ports or to none at all.

This flexibility lets us change the design of the Dining Philosophers a bit. In the current implementation each philosopher has lots of state (it knows its left and right chopstick queues, its join and the graph that contains it.) Because function_nodes have only one output, which is always sent to, each philosopher had to explicitly put the chopsticks back to the proper queues. In the original version of Dining Philosophers the join node for each philosopher was created and added to the graph after they finish thinking the first time.

The new version of the philosopher is simpler. The philosophers now only know about thinking, eating and putting chopsticks back.

The structure of the new graph is:

How it works:

1. The philosopher thinks (the first function_node), then emits a continue_msg.
2. The reserving join checks if it has an item at each input. If it does (the continue_msg plus the left and right chopstick), it assembles them into a std::tuple and forwards it to the eat() multioutput_function_node.
3. The multioutput_function_node eats, then puts its chopsticks back to their queues. If the philosopher has not completed M (currently 10) rounds of eating and thinking, it sends a continue_msg to the think() function_node. If the philosopher has completed M rounds, it does not emit the continue_msg, and no longer thinks or eats.

The graph is constructed with N philosophers, and then a continue_msg is sent to each think() function_node to start the graph. Here is the code for the main loop:

// create queues of (one) chopstick
chopstick_places_vector_type places(num_philosophers, tbb::flow::queue_node(g));
for ( int i = 0; i < num_philosophers; ++i ) {
    places[i].try_put(chopstick());
}

p_vector philosophers;
// must reserve the vector so no reallocation occurs (we're passing references to the vector elements)
philosophers.reserve(num_philosophers);
for ( int i = 0; i < num_philosophers; ++i )  philosophers.push_back( philosopher( names[i] ) );
    // push_back allowed because the philosopher objects can be assigned

// create think function_nodes
think_node_vector_type think_nodes;
think_nodes.reserve(num_philosophers);
for(int i=0; i < num_philosophers; ++i)
    think_nodes.push_back(new think_node_type(g, tbb::flow::unlimited, think_node_body(philosophers[i])));

// done queues (holds continue_msg)
thinking_done_vector done_vector(num_philosophers, thinking_done(g));

// create join nodes
join_node_vector_type j_vector(num_philosophers,join_node_type(g));

// attach chopstick buffers and think function_nodes to joins
for(int i = 0; i < num_philosophers; ++i) {
    think_nodes[i]->register_successor(done_vector[i]);
    done_vector[i].register_successor(tbb::flow::input_port<0>(j_vector[i]));
    places[i].register_successor(tbb::flow::input_port<1>(j_vector[i])); // left chopstick
    places[(i+1) % num_philosophers].register_successor(tbb::flow::input_port<2>(j_vector[i]));  // right chopstick
}
// create eat multioutput_function_nodes
eat_node_vector_type eat_nodes;
eat_nodes.reserve(num_philosophers);
for(int i = 0; i < num_philosophers; ++i) {
    eat_nodes.push_back( new eat_node_type(g, tbb::flow::unlimited, eat_node_body(philosophers[i])));
    // attach join to mofns
    j_vector[i].register_successor(*(eat_nodes[i]));
    // attach mofns to think function_nodes
    tbb::flow::output_port<0>(*(eat_nodes[i])).register_successor(*(think_nodes[i]));
    // attach mofns to chopstick queues
    tbb::flow::output_port<1>(*(eat_nodes[i])).register_successor(places[i]);
    tbb::flow::output_port<2>(*(eat_nodes[i])).register_successor(places[(i+1) % num_philosophers]);
}

// start all the philosophers thinking
for(int i = 0; i < num_philosophers; ++i) think_nodes[i]->try_put(tbb::flow::continue_msg());

g.wait_for_all();

for(int i = 0; i < num_philosophers; ++i) {
    delete think_nodes[i];
    delete eat_nodes[i];
}

There are a couple points to note:

• A queue_node (created as a vector of queue_nodes) is needed between the think() function_node and the reserving join, because the function_node is not reservable (if no successor accepts its output, that output is dropped.) The queue_node is reservable. (The reserving join does not buffer at its input ports, because if the join does not accept an input it must be available for other nodes to take.)
• Both the think() function_node vector (allocated at line 14) and the eat() multioutput_function_node vector (at line 33) must be pointers to the nodes. This is because flow::graph nodes are generally not assignable, and a push_back() to a vector involves an assignment. The nodes which do not need to be constructed one-by-one (the queues at lines 2 and 20, and the join at line 23) can be created by the std::vector constructor, because it uses copy-construction of its elements.

The multioutput_function_node is a vital addition to TBB flow::graph; I have found I use it in most of the graphs I have written since its debut.

Intel® TBB provides four graph nodes that do unbounded buffering. They are:

• buffer_node: this type of node simply buffers items, and hands them to receivers in no particular order.
• queue_node: this type of node buffers items, and hands them to receivers in FIFO order.
• priority_queue_node: this type of node buffers items that have a priority value associated with them, and hands them to receivers in highest-priority-first order.
• sequencer_node: this type of node buffers items that have a sequence number associated with them, and hands them to receivers in sequence number order; while the next item in the sequence is unavailable, the try_get operation fails.

Another possible default sending behavior of some nodes is that a data item is broadcast to multiple receivers, and if multiple receivers are available, this effectively duplicates the data item. We may require that every piece of data generated by a sending node be handled by at most one receiving node. In this case, we can place a single buffering node after the sender to prevent the duplication of data items.

Connecting buffering nodes with other graph nodes

If we connect a buffering node as a successor to another graph node, the buffering node always accepts anything that is sent to it. Thus, the underlying behavior is that the predecessor to the buffering node calls try_put on the buffering node whenever it has output to send. In other words, the predecessor always pushes data to the buffering node. The buffering node never goes into a pull state, that is, it never needs to call try_get on its predecessor.

On the output side of buffering nodes, the nature of the relationship between the buffering node and its successors can alternate between the push and pull states over time. When successors to a buffering node are idle, they are registered as successors of the buffering node, so that when the buffer receives content, the buffering node pushes that content to a waiting successor with a try_put operation. If all successors to a buffering node are busy, they may not be registered with the buffering node. In this case, when a successor becomes free, it tries to pull content from the buffering node via a try_get operation. Should this operation fail, the successor registers itself with the buffering node so that it can receive content later when it is available. More detail on alternating push/pull behavior is available in Mike Voss’ s blog here.

The behaviors described above are the implicit behavior of connections made with buffering nodes via the make_edge function. Other nodes may also explicitly call try_put and try_get on buffering nodes to transfer content. They can even make and sever temporary connections to the buffering nodes using the register and remove operations described in the reference manual. When such connections are made, the buffering nodes assume their role as described above: as an always-accepting successor and as a changeable push/pull predecessor as determined by content availability.

Making reservations on buffering nodes

Each of these nodes allows reservation: a successor to a buffering node can reserve an item and decide whether or not to use it later. The procedure for reservation from the receiver’s perspective is the same for all buffering nodes: a receiver calls try_reserve on its predecessor, and if no item is available, the reserve operation fails. If a reservable item is available, it gets a copy of the item to examine. Later, it may release the reservation by calling try_release, in which case the item stays in the buffer and is made available to other receivers, or it may consume the reservation by calling try_consume, in which case the item is removed from the buffer. From the buffering node’s perspective there are a few differences:

• On a reserve, the buffer_node sets aside an item (not necessarily the same item as if a try_get was called instead of a try_reserve), and continues processing other try_get operations from other receivers, handing out non-reserved items from the buffer.
• queue_nodes and priority_queue_nodes reserve their first or highest priority item, respectively. Get operations fail while a reservation is held.
• sequencer_nodes check if the item with the next sequence number is available and if so, it is reserved. If not, the reserve operation fails. Here also, get operations fail while a reservation is held.

Example

To illustrate the usage of buffering nodes, we provide a simple example that addresses the binpacking problem. For this problem, we are given N objects with weights v1,…,vN. The problem is to pack the objects into as few bins as possible, where each bin has capacity V. This is an NP-hard problem, but there are many ways to achieve near-optimal solutions.

For our implementation, we used a source_node to generate the N objects, and these are placed in a value pool (a queue_node) where they wait until they can be processed. The bulk of the algorithm is carried out by concurrent bin packers (function_nodes). Each bin packer is serial, and is attempting to pack its own bin using a best-fit greedy relaxation approach. These nodes have three output modes: they may send out a completely or partially filled bin to a bin buffer (buffer_node), where the bin waits to be accounted for in a final phase, or they may reject the object that was received, returning it to the value pool, or they may output nothing, which implies that the object was added to the bin, but the bin is not yet ready. Bins sent to the bin buffer are picked up by a final function_node that may output bin information and compile a summary of all the bins. The graph for this binpacking algorithm thus looks like this:

The choice of a queue_node for the value pool was motivated by the fact that we did not want a bin packer to reject an object by putting it back into the pool, only to pick it up again immediately. Thus the FIFO ordering in queue_node would reduce the likelihood of this. The bin buffer is simply a buffer_node since the order in which bins are collected in the final phase is irrelevant.

Below we show a code excerpt that sets up this graph. Note how the make_edge calls in the code below correspond to the solid black lines in the diagram. The implicit connections are made by calls to try_put within the algorithm used by the bin_packer nodes (code not shown here). The graph nodes that are created below use certain function objects to accomplish their goals. In particular, the_source uses an item_generator, bin_packers use bin_fillers, and the_writer uses bin_printer.

typedef size_t value_type; // the type of items we are packing
typedef vector < value_type > bin; // a bin is a vector of value_types
// Bin_packers are function_nodes that receive value_type items;
// they implicitly send packed bins to the_bin_buffer, and return
// unused value_type items back to the_value_pool:
typedef function_node < value_type > bin_packer;
// the_value_pool is represented by a queue_node:
typedef queue_node < value_type > value_pool;
// the_bin_buffer is represented by a buffer_node:
typedef buffer_node < bin > bin_buffer;
// the_writer is represented by a function_node:
typedef function_node < bin, bin > bin_writer;
// the_source is represented by a source_node:
typedef source_node < value_type > value_source;
bin_packer **bins; // the array of bin packers
...
graph g;
value_source the_source(g, item_generator(), false);
value_pool the_value_pool(g);
make_edge(the_source, the_value_pool);
bin_buffer the_bin_buffer(g);
bins = new bin_packer*[num_bin_packers];
for (size_t i=0; i
bins[i] = new bin_packer(g, 1,
bin_filler(i, &the_value_pool, &the_bin_buffer));
make_edge(the_value_pool, *(bins[i]));
}
bin_writer the_writer(g, 1, bin_printer());
make_edge(the_bin_buffer, the_writer);
the_source.activate();
g.wait_for_all();