The hybrid push-pull protocol used by tbb::graph biases communication to prevent polling and to reduce unnecessary retries.  Understanding the details of this protocol is not necessary to use tbb::graph, but it makes understanding its performance easier. 

Nodes in a graph are persistent and exist until a user explicitly destroys them. But unlike some actor systems, a thread is not assigned to each tbb::graph node.  Tasks are created on-demand to execute node bodies and pass messages between nodes when there is activity in the graph.   Consequently, a tbb::graph node does not spin in a loop waiting for messages to arrive.  Instead when a message arrives, a task is created to apply the receiving node’s body to the incoming message.

If nodes always accept incoming messages, this is straightforward to implement.  Each time a message arrives at a node, a task can be created to apply the body to that message and to forward the result to the node’s successors.  However some nodes, such as function_nodes or limiter_nodes, can reject an incoming message.  For example, a function_node will reject an incoming message if it has reached its maximum allowable concurrency.  

The challenge is to create a protocol for dealing with message rejection that is efficient and ensures that messages aren’t accidentally dropped.   To be efficient in a non-preemptive tasking system like Intel® Threading Building Blocks, it’s important to not create many small tasks or tasks that waste resources by spinning.  Creating a new task to retry at each rejection may generate many small useless tasks, and repeatedly issuing sends in a loop until one is accepted is likewise inefficient.

Instead, tbb::graph uses a hybrid push-pull protocol as a more efficient alternative.

Figure 1: A state diagram for the hybrid push-pull protocol used by tbb::graph.

In the state diagram shown in Figure 1, edges dynamically switch between a push and pull protocol at rejections.  An Intel® TBB graph G = ( V, S, L ), where V is the set of nodes, S is the set of edges that are currently using a push protocol, and L is the set of edges that are currently using a pull protocol.  For each edge (Vi, Vj), Vi is the predecessor / sender and Vj is the successor / receiver.  When in the push set S, messages over an edge are initiated by the sender, which tries to put to the receiver.  When in the pull set, messages are initiated by the receiver, which tries to get from the sender.  If a message attempt across an edge fails, the edge is moved to the other set.  For example, if a put across the edge (Vi, Vj) is rejected, the edge is removed from the push set S and placed in the pull set L.

This protocol results in a reduction in the messages across an edge, while maintaining quick response times.  If a sender, Vi, generates data at a faster rate than its successor Vj, the edge will transition into pull mode, eliminating the many rejections that Vi would see if it were to continue to send.   Likewise, if a receiver Vj processes data faster than its sender Vi, the edge will stay in push mode, allowing Vi to send data as soon as it is generated.

There are two interesting scenarios in a tbb::graph program when considering this protocol: (1) when there is no buffering between the sender and receiver and (2) when there is buffering between the sender and receiver.   Using tbb::graph, there are two ways that one can deal with a node that rejects messages.  If no buffering is placed before the node, rejected messages will be dropped.  If buffering is placed before the receiver, then messages will be buffered until the receiver can consume them.  The protocol in Figure 1 works in both cases to reduce unnecessary rejections.

Let’s first consider the unbuffered case, as shown in Figure 2 below.

Figure 2: A simple sub-graph with two function_nodes.

In Figure 2, we have a function_node, f1, sending its output to a function_node, f2.   Node f1 has unlimited concurrency, which allows its body to be applied concurrently to multiple inputs. Node f2, however, is restricted to a concurrency of 1.    If f2 is busy applying its body to another message when a new message arrives from f1, it rejects the incoming message.  Without a buffer between the two nodes, the message is dropped.  Admittedly, it is rare that a user will want messages to be dropped.  In fact, in an upcoming update of the graph API, buffering will be added by default to the input of a function_node if it has limited concurrency.   After that update, users will have to explicitly choose the dropping behavior.

In any case, let’s say that f2 in Figure 2 rejects a message sent by f1.  Because of the protocol in Figure 1, the edge f1->f2 is changed so that f1 will no longer put to f2 and instead f2 must pull from f1.  This does not mean that f1 will not apply its body to messages as they arrive at f1, but just that it will no longer send the results of its body to f2.   If f1 had other successors that did not reject the message, they would continue to receive subsequent results.

When f2 finishes executing its body, it becomes free to process new messages and tries to pull from f1.  Because a function_node does not have input or output buffering, it can never produce output on demand.  Thus, it always rejects attempts to pull from it.  This rejection returns the edge to push mode, and subsequently f1 will again push to f2. 

So what has this accomplished?  If f1 and f2 are imbalanced, these switches, from push to pull and then back to push, reduce the number of failed puts sent from f1 to f2.  While f2 is busy, f1 stops sending messages that will ultimately be rejected.  Only when f2 becomes free is the edge re-established for pushing.

The more common case, and what will become the default in an upcoming update, is when there is buffering between the two nodes as shown in Figure 3.

Figure 3: A simple sub-graph with two function_nodes and queue between them.

In this case, f1 puts its result to q1, which always accepts.  In turn, q1 attempts to forward items to f2 in first-in first-out order.  If f2 rejects an item, it remains at the head of the queue in q1, until it can be successfully passed to f2.  In this example, the edge f1->q1 will always stay in push mode, since q1 does not reject.  However, the edge q1->f2 may change since f2 can reject if it is busy.

Again, let’s assume that at times items arrive in q1 at a faster rate than they can be consumed by f2.  If f2 rejects a send from q1, the edge q1->f2 is transitioned to the pull state.  Because q1 buffers rejected items, this message is not lost.  When f2 finishes executing its body, it tries to pull from q1.  Since there is an item buffered in q1, the pull succeeds and the item is given to f2.  Since the pull was successful, the edge q1->f2 stays in pull mode.  When f2 finishes execution on the item it has just pulled, it will pull again from q1.  It will continue to do so, until q1 is empty and therefore rejects f2’s pull request.

As with the previous scenario, the switching of modes from push to pull reduces the number of messages across the edge.  Only when f2 is available to do work does it make a request for an item from q1, thereby removing the need for q1 to periodically poll or send items speculatively.

This example will demonstrate:

• How to use the graph's run function.
• How to mix explicit puts with explicit edges
• The non-greedy nature of the join_node

In this post, I'll provide an implementation for the Dining Philosophers problem shown below.  In this problem, several philosophers are sitting together at a table.  Each philosopher needs to both think and eat, but can only do one of these at a time.  They each think, eat, think, eat, etc...  In the figure below, the philosophers are using chopsticks to eat noodles.  They must grab both the chopstick to their left and the chopstick to their right before eating. To complicate things, the chopsticks are shared with their neighbors.  So a philosopher's left chopstick is their left neighbor's right chopstick.  And their right chopstick is their right neighbor's left chopstick.  

Dining Philosophers is a challenging problem because it will deadlock without proper cooperation between the philosophers.  For example, if all of the philosophers start by grabbing their left chopstick, then there will be no right chopstick available for any of them.  None of them will be able to eat (and subsequently think) unless their right neighbor gives up the chopstick they have already claimed.  There are a number of existing solutions to the Dining Philosophers problem.

I'll use the tbb::graph and its associated node classes to implement a solution to Dining Philosophers.  In my solution, each philosopher will be an object that contains a join_node that will capture the chopsticks and a function_node that will perform the eating and thinking. The chopsticks will be null objects and their places on the table will be implemented as queue_nodes. If a queue_node has an item, it means that the chopstick is available at that place, otherwise it is not available. At most each queue_node will contain one item.  The graph for 4 philosophers will therefore be structured like the figure below.

As with all Community Preview Features, the graph must be explicitly enabled. This is done by defining its macro, TBB_PREVIEW_GRAPH, before including the header file as shown below.

#define TBB_PREVIEW_GRAPH 1
#include "tbb/graph.h"

The main function is shown below. 

const char *names[] =
{ "Archimedes", "Aristotle", "Democritus", "Epicurus", "Euclid",
"Heraclitus", "Plato", "Pythagoras", "Socrates", "Thales" };

int main(int argc, char *argv[]) {
  int num_threads = 0;
  int num_philosophers = 10;
  if ( argc > 1 ) num_threads = atoi(argv[1]);
  if ( argc > 2 ) num_philosophers = atoi(argv[2]);

  if ( num_threads < 1 || num_philosophers < 1 || num_philosophers > 10 ) exit(1);

  tbb::task_scheduler_init init(num_threads);
  tbb::graph g;
  printf("\n%d philosophers with %d threads\n\n",
         num_philosophers, num_threads);

  std::vector< tbb::queue_node * > places;
  for ( int i = 0; i < num_philosophers; ++i ) {
    tbb::queue_node<chopstick> *qn_ptr = new tbb::queue_node(g);
    qn_ptr->try_put(chopstick());
    places.push_back( qn_ptr );
  }

  std::vector< philosopher > philosophers;
  for ( int i = 0; i < num_philosophers; ++i ) {
    philosophers.push_back( philosopher( names[i], g,
                                         places[i],
                                         places[(i+1)%num_philosophers] ) );
    g.run( philosophers[i] );
  }
  g.wait_for_all();

  for ( int i = 0; i < num_philosophers; ++i ) philosophers[i].check();

  return 0;
}

After the initial command line processing is done in the main function above, a graph object is instantiated. A vector places of queue_node<chopstick> pointers is then populated with queues that will represent the places at the table.

After each queue_node is created, a single chopstick object is put to it, indicating that a chopstick is initially available at that location.

After the queues are created, the main function then populates a vector of philosopher objects. After each philosopher is added to the vector, it is passed to the graph object's run function. As I will show shortly, class philosopher not only contains a function_node and join_node but it is also a function object, defining a void operator()(). The graph's run function executes this function object in a task that is a child of the graph's root task. No calls to g.wait_for_all() will return until all tasks that are children of this root task complete.  The philosophers use their operator() functions to think once and then insert themselves in to the graph. The main function ends by checking each philosopher object to verify that it has called think and eat the proper number of times.

There is also version of run that takes a second argument: template<typename Receiver, typename Body> void run( Receiver &r, Body body ). Like the version used in this example, it creates a task that runs body but also sends the value returned by body to the receiver r.

My declaration of class philosopher is shown below:

const int think_time = 1;
const int eat_time = 1;
const int num_times = 10;

class chopstick {};

class philosopher {
public:

  typedef tbb::queue_node< chopstick > chopstick_buffer;
  typedef tbb::join_node< chopstick, chopstick > join_type;

  philosopher( const char *name, tbb::graph &the_graph,
               chopstick_buffer *left, chopstick_buffer *right ) :
   my_name(name), my_graph(&the_graph),
   my_left_chopstick(left), my_right_chopstick(right),
   my_join(new join_type(the_graph)), my_function_node(NULL),
   my_count(new int(num_times)) {}

  void operator()();
  void check();

private:

  const char *my_name;
  tbb::graph *my_graph;
  chopstick_buffer *my_left_chopstick;
  chopstick_buffer *my_right_chopstick;
  join_type *my_join;
  tbb::function_node< join_type::output_type, tbb::continue_msg > *my_function_node;
  int *my_count;

  friend class node_body;

  void eat_and_think( );
  void eat( );
  void think( );
  void make_my_node();

};

Each philosopher has a const char *my_name that holds its name. It also has pointers to the graph, the two chopstick queues that it is seated near, its join_node, its function_node and the counter that it will use to track how many times its been called.

Let's first look at the definition of void operator()(), which is invoked by the tasks enqueued by calls to run in main. This function calls think and then make_my_node.  So each philosopher will first think and then afterwards insert itself into the graph.

void philosopher::operator()() {
  think();
  make_my_node();
}

Both function think and function eat (which will be used later) are straightforward functions that just sleep:

void philosopher::think() {
  printf("%s thinking\n", my_name );
  SLEEP(think_time);
  printf("%s done thinking\n", my_name );
}

void philosopher::eat() {
  printf("%s eating\n", my_name );
  SLEEP(eat_time);
  printf("%s done eating\n", my_name );
}

The function make_my_node is responsible for creating the function_node and connecting both the join_node and function_node to the rest of the graph. The join_node's input ports are stored in a std::tuple, which is returned by the call to inputs(). I use the template function std::get to access the needed element. The implementation of make_my_node is shown below:

void philosopher::make_my_node() {
  my_left_chopstick->register_successor( std::get<0>(my_join->inputs()) );
  my_right_chopstick->register_successor( std::get<1>(my_join->inputs()) );
  my_function_node =
    new tbb::function_node< join_type::output_type, tbb::continue_msg >( *my_graph,
      tbb::graph::serial, node_body( *this ) );
  tbb::make_edge( *my_join, *my_function_node );
}

The class node_body is a straightforward function object that invokes the corresponding philosopher's eat_and_think function.

class node_body {
  philosopher &my_philosopher;
public:
  node_body( philosopher &p ) : my_philosopher(p) { }
  void operator()( philosopher::join_type::output_type ) {
    my_philosopher.eat_and_think();
  }
};

The implementation of eat_and_think(), calls the philosopher's function eat and then decrements its count. If the philosopher stills needs to eat and think more, then it puts its chopsticks back down on the table and thinks. Otherwise, it removes its join_node from the graph before putting its chopsticks back down on the table.

void philosopher::eat_and_think( ) {
  eat();
  --(*my_count);

  if (*my_count > 0) {
    my_left_chopstick->try_put( chopstick() );
    my_right_chopstick->try_put( chopstick() );
    think();
  } else {
    my_left_chopstick->remove_successor( std::get<0>(my_join->inputs()) );
    my_right_chopstick->remove_successor( std::get<1>(my_join->inputs()) );
    my_left_chopstick->try_put( chopstick() );
    my_right_chopstick->try_put( chopstick() );
  }
}

The code above demonstrates that nodes can be connected by explicit edges, as is the case for the queue_nodes and the join_node. And user code can also do explicit try_puts to nodes. In this example, there is no explicit edge from the philosopher back to its chopstick queues. However, eat_and_think explicitly calls try_put to put chopstick objects in to the queues.

Finally at the end of main, each philosopher's function check is called to verify that it has been executed the correct number of times (and it also does some cleanup).

void philosopher::check() {
  if ( *my_count != 0 ) {
    printf("ERROR: philosopher %s still had to run %d more times\n", my_name, *my_count);
    exit(1);
  } else {
    printf("%s done.\n", my_name);
  }
  delete my_function_node;
  delete my_join;
  delete my_count;
}

When I execute this example using four philosophers and a single thread, "philosophers 1 4", it runs in about 80 seconds. This is 4 x ( 10 thinks + 10 eats ) = 80. When I run it using all 8 threads available on my desktop, it completes in about 21 seconds.

The reason this example works at all is because of the non-greedy nature of the join_node. A join_node creates a std::tuple from the items it receives at its input ports. However, it does not greedily consume items as they appear. Instead, once it has received notification that an item is available at each port it then attempts to reserve each of these items. If it is successful, only then does it create the tuple and consume the items. If it cannot reserve an item at any one port, it releases all reservations it has previously made.

In the Dining Philosopher's problem, the join_node prevents deadlock by never holding a chopstick unless it can acquire both. It may reserve one of the chopsticks, but if it cannot reserve the other, it puts the first one back on the table and tries again.

This example will calculate the sum of x*x + x*x*x for all x = 1 to 10. This is a simple syntactic example only. Since each node in a graph may execute as an independent task, the granularity of each node should follow the general guidelines for tasks as described in Section 3.2.3 of the Intel® Threading Building Blocks Tutorial. But for demonstration purposes, I will use an artificial, tiny example here and inflate the time spent in each node by sleeping for 1 second after each operation.  I use the Linux function sleep for this. If you want to enter this example yourself, you'll have to use the appropriate sleep function for your system.

The basic layout of the graph that I’ll create is shown in the figure below. Each value enters through the input node. This node will broadcast the value to both squarer and cuber, which will calculate x*x and x*x*x respectively (and sleep for 1 second). The output of each of these nodes will be placed in an unbounded buffer. A tuple containing both values will be created by the join node and forwarded to summer, which will add both values to the running total (and sleep for 1 second). The squarer and cuber will allow unlimited concurreny, that is they will be allowed to process multiple values simultaneously. The final summer, which updates a shared total, will be only allowed process a single in-coming tuple at a time, eliminating the need for a lock around the shared value.

As with all Community Preview Features, the graph must be explicitly enabled. This is done by defining its macro, TBB_PREVIEW_GRAPH, before including the header file as shown below.

#define TBB_PREVIEW_GRAPH 1
#include "tbb/graph.h"

This example performs three basic types of operations: square, cube and sum. The classes below define these operations and will be use as the body objects for my function_nodes.

struct square {
  int operator()(int v) {
    printf(“squaring %d\n”, v);
    sleep(1);
    return v*v;
  }
};

struct cube {
  int operator()(int v) {
    printf(“cubing %d\n”, v);
    sleep(1);
    return v*v*v;
  }
};

class sum {
  int &my_sum;
public:
  sum( int &s ) : my_sum(s) {}
  int operator()( std::tuple<int,int> v ) {
    printf(“adding %d and %d to %d\n”, std::get<0>(v), std::get<1>(v), my_sum);
    my_sum += std::get<0>(v) + std::get<1>(v);
    return my_sum;
  }
};

In function main, the graph is setup and then the values 1 – 10 are put into the input node. All the nodes in this example pass around values of type int. The nodes used below are all class templates and therefore can be used with any type that supports copy construction, including pointers and objects. It should be noted that values are copied as they pass between nodes, so passing around large objects should be avoided.

using namespace tbb;

int main() {
  int result = 0;

  graph g;
  broadcast_node<int> input;
  function_node<int,int> squarer( g, graph::unlimited, square() );
  buffer_node<int> square_buffer(g);
  function_node<int,int> cuber( g, graph::unlimited, cube() );
  buffer_node<int> cube_buffer(g);
  join_node<int,int> j( g );
  function_node<std::tuple<int,int>,int> summer( g, graph::serial, sum(result) );

  make_edge( input, squarer );
  make_edge( input, cuber );
  make_edge( squarer, square_buffer );
  make_edge( square_buffer, std::get<0>( j.inputs() ) );
  make_edge( cuber, cube_buffer );
  make_edge( cube_buffer, std::get<1>( j.inputs() ) );
  make_edge( j, summer );

  for (int i = 1; i <= 10; ++i)
    input.try_put(i);
  g.wait_for_all();
  printf("Final result is %d\n", result);
  return 0;
}

Towards the top of the code above, the graph nodes are created: 1 broadcast_node, 3 function_node objects , 2 buffer_node objects, and 1 join_node. After the nodes are created, they are linked together using make_edge calls. Both the nodes and edges correspond directly to the figure presented earlier.

Once the graph has been setup, the values 1-10 are put into the input node. The wait_for_all call will block until there is no more activity in the graph. The result of the computation is then printed to stdout and should look something like:

cubing 10
squaring 1
cubing 1
squaring 2
cubing 2
cubing 3
squaring 4
squaring 3
squaring 5
cubing 5
cubing 4
squaring 6
cubing 6
squaring 7
cubing 7
squaring 8
squaring 10
adding 1 and 1000 to 0
cubing 8
squaring 9
cubing 9
adding 4 and 1 to 1001
adding 9 and 8 to 1006
adding 25 and 27 to 1023
adding 36 and 125 to 1075
adding 49 and 64 to 1236
adding 64 and 216 to 1349
adding 16 and 343 to 1629
adding 100 and 512 to 1988
adding 81 and 729 to 2600
Final result is 3410

One might note that most of the square and cube operations happened before any of the sum operations in the above run. This is an artifact of using a for loop to inject the values 1-10 into the graph. In a future example, I'll demonstrate how a source_node can be used to create a better execution ordering.

 In this example, I setup five computations A-E, and enforce the partial ordering shown in the figure below. For each edge in the graph, the node at the tail of the edge must complete its execution before the node at the head may begin.

As with all Community Preview Features, the graph must be explicitly enabled. This is done by defining its macro, TBB_PREVIEW_GRAPH, before including the header file as shown below. 

#define TBB_PREVIEW_GRAPH 1
#include "tbb/graph.h" 

To keep this example simple, I make the bodies of the nodes just print out their names and then sleep for 1 second. I use the Linux function sleep for this. If you want to enter this example yourself, you'll have to use the appropriate sleep function for your system. All of these nodes are able to use the class body below. 

#include <cstdio>

struct body {
  std::string my_name;
  body( const char *name ) : my_name(name) {}
  void operator()( continue_msg ) const {
    printf("%s\n", my_name.c_str());
    sleep(1);
  }
};  

In function main, the graph is setup once and then run three times. All of the nodes in this example pass around continue_msg objects. This type is defined in graph.h and is used to communicate that a node has completed its execution. 

using namespace tbb;

int main() {
  graph g;
  broadcast_node< continue_msg > start;
  executable_node< continue_msg > a( g, body("A"));
  executable_node< continue_msg > b( g, body("B"));
  executable_node< continue_msg > c( g, body("C"));
  executable_node< continue_msg > d( g, body("D"));
  executable_node< continue_msg > e( g, body("E"));
  make_edge( start, a );
  make_edge( start, b );
  make_edge( a, c );
  make_edge( b, c );
  make_edge( c, d );
  make_edge( a, e );
  for (int i = 0; i < 3; ++i ) {
    start.try_put( continue_msg() );
    g.wait_for_all();
  }
  return 0;
} 

The first line in function main instantiates a graph object g. On the next line, a broadcast_node named start is created. Anything passed to this node will be broadcast to all of its successors. The start node is used in the for loop at the bottom of main to launch the execution of the rest of the graph. 

In the code above, five executable_node objects are created, named a – e. Each node is constructed with a reference to graph g and the function object to invoke when they run. The successor / predecessor relationships are setup by the make_edge calls that follow the declaration of the nodes.

After the nodes and edges are setup, the try_put in each iteration of the for loop results in a broadcast of a continue_msg to both a and b . Both a and b are waiting for a single continue_msg, since they both have only a single predecessor, start. 

When they receive the message from start, they execute their body objects. When complete, they each forward a continue_msg to their successors, and so on. The graph uses Intel® Threading Building Blocks tasks to execute the node bodies as well as forward messages between the nodes, allowing computation to execute concurrently when possible. 

If you run this example on a system with at least two threads, the output for each iteration of the loop will look something like: 

A
B
(pause)
E
C
(pause)
D 

The (pause) above is not shown in the output, but represents a noticeable (1 second) delay. 

The critical path in this example is {start, B, C, D} which has a length of 3 seconds. When run on a systems with more than 1 hardware thread, the execution of the critical path can be overlapped with the executions of nodes A and E. The total time to perform the three executions of the graph will therefore take roughly 9 seconds, or 3 seconds per iteration. 

For additional information about the graph and its nodes, you can read Appendix D in the Intel® Threading Building Blocks Reference Manual. I will also be posting additional examples that highlight other key features of the graph over the coming weeks.

There are three types of components used to implement a graph: a graph object, nodes and edges.

A user first creates a graph object. This object acts as the owner of all tasks created on behalf of the graph and its nodes. In many ways its like a task_group, and users can call wait_for_all on the graph object if they need to wait for all of the tasks related to the graph to complete.

Next users create the nodes and edges. Nodes process or buffer messages as they pass through them. There are some node types that invoke user-provided function objects, while others direct, buffer or combine messages in a pre-specified way. The user creates nodes and then links them together by calling the functions make_edge or make_edges.

In future posts over the coming weeks, I will provide several example applications that highlight key features of the graph and its node types. A detailed description of class graph and the different node types are provided in Appendix D of the Intel® Threading Building Blocks Reference Manual.

As with all Community Preview features, the graph must be explicitly enabled by setting its macro before including its header:

#define TBB_PREVIEW_GRAPH 1
#include "tbb/graph.h"

Below are brief descriptions of the node types currently provided in graph.h. In future posts, I will use these nodes to construct several example applications.

The first three node types invoke user-provided function objects and are used to wrap user computations:

template < typename OType > class source_node;
A source_node has no predecessors in the graph. It generates messages of a generic type OType by calling a user-provided body object. The generated message is broadcast to all of its successors.

template < typename IType, typename OType > class function_node;
A function_node receives messages of a generic type IType, applies a body object to each message, and broadcasts its output of type OType to its successors in the graph.

template < typename OType > class executable_node;
An executable_node receives messgaes of a type continue_msg. When it has received a continue_msg from each of its predecessors, it invokes its body object to generate a message of type OType that is passes to its successors in the graph.

The remaining 10 node types all perform pre-defined operations that buffer, direct or combine messages:

class contine_node;
A continue_node receives continue_msg messages. When it receives a message from each of its predecessors, it forwards a single continue_msg to all of its successors.

template < typename T > class broadcast_node;
A node that broadcasts each incoming message to all of its successors.

template < typename T > class write_once_node;
A write_once_node buffers a single item that cannot be over written. The value may be cleared explicitly, after which a new value may be set.

template < typename T > class overwrite_node;
An overwrite_node buffers a single item that can be over written.

template < typename T > class buffer_node;
A buffer_node is an unbounded buffer of messages of type T. Messages are forwarded in arbitrary order.

template < typename T > class queue_node;
A queue_node is an unbounded buffer of messages of type T. Messages are forwarded in first-in first-out order.

template < typename T > class priority_queue_node;
A priority_queue_node is an unbounded buffer of messages of type T. Messages are forwarded in priority order.

template < typename T > class sequencer_node;
A sequencer_node is an unbounded buffer of messages of type T. Messages are forwarded in sequence order.

template < typename T > class limiter_node;
A limiter_node limits the number of messages that may pass through it. It contains an embedded continue_node that may be used to decrement the count to allow additional items through.

template < typename T1, typename T2, … > class join_node;
A join_node creates a std::tuple<T1,T2,…> from a set of messages received at its inputs.