Hello. I am interested in experimenting with modifications to the libcilkrts implementation in gcc for Linux. Specifically, I would like to attach scheduling priorities to spawned work. For the sake of simplification, we can assume that these are SCHED_FIFO priorities. (In truth, priorities will be deadline-based in the LitmusRT Linux kernel. LitmusRT is a real-time extension to Linux used in university research (www.litmus-rt.org).)
Suppose I wish to attach a SCHED_FIFO priority of 5 to some spawned work such that the worker thread that processes this work is scheduled with a priority of 5. I could pass this as a regular argument in CilkPlus and let the worker thread call pthread_setschedprio() to set the proper priority (and restore its old priority upon completion). However, there is a condition where the worker thread may never be able to wake up to call pthread_setschedprio(): Suppose our waiting worker thread has a default priority of 0 and we want to send priority-5 work to it. However, at the time this work is spawned, the CPUs are all occupied with threads running at priority 3. The newly spawned work *should* preempt the priority-3 threads, but it can't since the worker thread's initial priority is 0---the thread is blocked from waking and cannot call pthread_setschedprio().
I can think of two possible workarounds:
1) After CilkPlus is initialized, set all worker thread priorities to 99 (maximum priority). These threads won't consume any CPU time while they are waiting for work. However, they will always be able to run when woken up to process new work. They can then *downgrade* their priority as outlined above and restore their default 99 priority when work is completed.
2) Let the thread that spawns the work set the scheduling priority of the worker thread that will process the work.*
I am happy to use either technique, but I would prefer #2 if the notifications of new work to worker threads is broadcasted. I don't want to trigger a flood of priority-99 threads to process one unit of work. This would needlessly preempt other threads on the system.
Is there a design document that discusses the design of libcilkrts? I have read the Cilk-5 paper from '98, but it appears to focus on work stealing. If there is no such document, can someone walk me through the calls that are taken in libcilkrts when new work is spawned? I've studied the libcilkrts code (including scheduler.c) and stepped through code in gdb, but I haven't yet identified the segment of code that says, "Here is some new work. Wake up a worker to handle it." I think scheduler.c::notify_children() might be what I want, but the context in which it is called doesn't quite match my expectations.
* Under either method, #1 or #2, I plan to set CILK_NWORKERS to be great enough to ensure that there is always an idle worker thread. This will allow later-arriving higher-priority work to preempt lower-priority work already in-flight. I know that this obviates the work stealing features of CilkPlus, but I want to put scheduling decisions exclusively in the domain of the operating system.