Hi, so, all the nodes now have 40 cores, great :D but I still want to know if when I state "#PBS -l select=1:ncpus=32", but my application launches 64 threads, what is going to happen? (1) all 40 cores are going to be used (2) the submission system in conjunction with the OS is smart enough to make use of ONLY the 32 cores that specified. thanks, Daniel Nicacio
Your thread count and PBS ncpus are totally independent.
The PBS ncpus just requestsa specific number ofcpu resources from the batch node:
* On a shared node (acano02), leaving any CPU resources on the table (i.e. requesting less than 40), will allow others to pick up the remaining cpu resources for their batch jobs.
* On the remaining exclusive nodes (acano03 and acano04), running with any number of ncpus will get exclusive access to the whole machine.
So I'd recomend that if you plan to use a thread count >40, then you should always specify ncpus=40.
Note: the batch nodes have Hyperthreading turned off.
Note: you may wish to use taskset(1) tobond threads to cores, PBS doesn't do this.
Hi Mike, thanks for the prompt reply. I was asking this beacause I have done some experiments in the past using the 32 core batch nodes, so in order to not have to run every thing again, I was trying to find a way to limit the number os cores to be used to 32. If you know any way to do that, please let me know. But running everything again and see how it behaves with 40 cores is not a bad idea too. Thanks again, Daniel Nicacio
I would suggest that you use taskset(1) to assign/limit your workload to the similar cores that were on the 32-core MTL. We now have 10-cores per socket instead of 8-cores, so it should be possible to use taskset(1) to use only 8-cores per socket. See /proc/cpuinfo and the taskset(1) man page orMTL getting Started Guidefor details on howto dothe allocation.