using TBB to emulate process scheduler?

memaher

Wed, 03/21/2012 - 03:09

Morning All,

I've got an application which needs to do some rather funky process scheduling. This is definitely not what TBB is designed for, but it seems a rather close match. Posting here partially to see if TBB could adapt to this application, and partially to glean some interesting ideas! Let me explain...

We've got a large application framework running under Linux on multi-Xeon workstations using TBB. I've been asked to spec-up an enhancement to this application which is essentially to emulate some low-speed embedded systems, running RTOS. The use of traditional emulators/VM's is probably out the question as they need to integrate into the current framework. Luckily all the embedded applications are written in c-code, so a direct recompile is possible (first frigging the h/w accesses into our framework).

My immediate thought was to do a direct recompile of the RTOS applications onto Linux, sort the hardware I/O out, then slug'em. Code-wise its possible as the underlying O/S is posix compliant. The big problem is that we're taking 17MHZ single-cpu applications and running them on a 12-core 2.4ghz cpu - the race conditions would be horrendous as all applications start executing in parallel.

The only viable alternative would be to execute all RTOS applications as threads/tasks under a single linux application, with some kind of scheduler/CPU management going on to enforce linear operation. This is where TBB comes in. My initial thought was to allocate each application into a TBB pipeline, adding slug/delay tasks in between.

The real gotcha with this idea is that these applications are designed to be suspended by the underlying O/S, which won't happen if they're turned into tasks - I'd have to insert manual break-points into their code to allow fair scheduling.

Ultimately, taking this idea one step forward: is there any way in TBB to automatically break and re-schedule tasks? For example if we had a tasks with infinite duration, asking TBB to automatically suspend operation and move onto the next task, with operation to be picked up later. Essentially replicating the feature of a basic kernel scheduler, but at a much higher application level?

As I said, I know this is definitely not what TBB has been designed for: I'm actually trying to use it to enforce single-core, slow operation!!!

Any thoughts/comments/answers/suggestions welcome!

Mat

6 posts / 0 new

Last post

For more complete information about compiler optimizations, see our Optimization Notice.

ahelwer

Wed, 03/21/2012 - 15:27

"A task in Intel Threading Building Blocks, in contrast, is typically a smallroutine, and also, cannot be preempted at the task level (though its logical thread canbe preempted)."

-Section 11.1, Task-Based Programming, Tutorial

However, there are definitely defined cases where tasks are preempted - see Continuation Passing, section 11.5.2 in the tutorial. Calling spawn_and_wait_for_all() inside a task will cause that task to be pulled off the thread it's executing on until its children complete. I'm not really sure how that's accomplished, but logically functionality must exist for storing the task's stack. This is really plumbing the depths of the scheduler though, so it would probably be easier to roll your own than mess around with this.

Raf Schietekat

Wed, 03/21/2012 - 16:03

"Calling spawn_and_wait_for_all() inside a task will cause that task to
be pulled off the thread it's executing on until its children complete.
I'm not really sure how that's accomplished, but logically functionality
must exist for storing the task's stack. This is really plumbing the
depths of the scheduler though, so it would probably be easier to roll
your own than mess around with this."
The task is not separated from its thread: the thread is just doing other things for a while as if it were executing a subroutine, until at some point it returns and picks up where it left off.

ahelwer

Wed, 03/21/2012 - 16:32

Ah yes, that would make a lot more sense.

memaher

Fri, 03/23/2012 - 09:30

so in theory, my emulator could load up very large tasks, then call spawn_and_wait_for_all() at regular discrete intervals, thus unloading the process, letting another task use that thread until it too calls spawn...()?

Hmmm, so the bigger question is:
- how would you resume the task once its been unloaded?
- is there any way to call this function from outside the task?

I know TBB isn't into threads per-say, but do you know if theres any way to externally suspend/resume threads upon command (i.e. manual override of the kernel scheduler?)

yeah, I know, I'm clutching at straws...

cheers

Mat

ahelwer

Fri, 03/23/2012 - 16:08

spawn_and_wait_for_all is called within the task, not externally.The main problem is there is no way to do any sort of asynchronous external interrupt of tasks with TBB.

Regarding the inner workings of spawn_and_wait_for_all, the only way for task A to resume after calling spawn_and_wait_for_all is for the tasks referenced by the spawn_and_wait_for_all call to finish and return. Task A's stack is just stored in place on the thread's stack, with the next scheduled task B's stack below it. So, task B must return before task A's stack space can be accessed (thus allowing task A to run again). Or so my understanding goes.

I guess you could shift preemption of tasks to the thread level. Set the affinity of all threads used by TBB to a single core, by implementing the virtual functions in tbb::task_scheduler_observer to set the affinity with normal OS API calls. Then call task_scheduler_init with parameter N, where N is the maximum number of threads you want to be time-sliced concurrently. Then your threads and the corresponding tasks they run will just be scheduled by the OS because you've oversubscribed the single core they're allowed to run on.

Back to original post

Please sign in to add a comment. Not a member? Join today

using TBB to emulate process scheduler?

Leave a Comment