I'm looking at TBB's task_arena and task_scheduler_observer.
The documentation for task_scheduler_observer sketches out a nice example of it being used to set thread affinity on worker threads to lock an arena's threads onto a subset of cores.
I'm curious to know whether this class and a similar pattern could practically be used to set OS scheduling priority for an arena. What I'm interested in doing is, on my N core HW, creating an arena with N normal worker threads, and another arena with N threads on a lower OS scheduling priority. However, the issue with scheduler priority is that generally you only get to lower it (unless running as root, but assume not), and it's not clear to me to what extent TBB worker threads move around between arenas (which would defeat the object of keeping all the low priority threads in one arena); the task_scheduler_observer docs mention returning false from on_scheduler_leaving() to keep a thread in an arena... but also mentions the possibility of it not being called if a thread is moved e.g for rebalancing. On the other hand in the affinity example if threads with a affinity mask set migrated out of the pinned arena into the general pool, that'd seem to be undesirable too but there doesn't seem to be any mention of it as an issue so maybe it's not a problem in practice for some reason?
BTW I've used two thread pools (homegrown ones) with different OS scheduler priorities in interactive applications before and found it useful for separating and prioritizing the tasks which are crucial to maintaining application responsiveness from those which aren't.