Hello Sergey, Do you have enough free memory and number of handles? It looks you have reached 2GB per process windows limitation. Could you work with 64 bit version to get more threads working? --Vladimir

Without using the BOOT.INI option to instruct the 32-bit Windows to permit processes to use up to 3GB of user space, the user application is limited to 2GB (plus system space in upper 2GB address range).

With 2GB
Subtract code size
Subtract static data
Subtract main thread initial stack
The remaining memory is in your initial heap
Prior to creating your threads you may perform allocations, remove this from the amount of available memory.

Assume for example you have 1GB remaining.

Default thread stack limit is 1MB. Therefore 1000 threads could possibly be created in the remaining 1GB assuming they used no additional resources. *** and leaving 0 RAM for additional allocations ***

64-bit does not have this limitation.

Does your system have more than 981 logical processors?
If not, then why so many threads???

Jim Dempsey

The (important) facilities of each OpenMP for thread affinity are limited to the number of logical processors with hardware support on the supported systems (no Intel platforms currently support more than 248 logical processors, and not on Windows).

Hello Sergey, try this example

#include

#include
DWORD WINAPI thread_routine(LPVOID lpParameter)

{

	Sleep(20000);

	return 0;

}
int main()

{

	unsigned int uiNumThreads = 0;

	HANDLE h[10000];
	for (uiNumThreads=1; uiNumThreads<10000; uiNumThreads+=1)

	{

		h[uiNumThreads]= CreateThread ( NULL, 0, (LPTHREAD_START_ROUTINE) thread_routine, (LPVOID)uiNumThreads, 0, 0 );

		if( h[uiNumThreads] == NULL ){

			printf( "Kernel object limit is %dn",uiNumThreads);

			break;

		}

	}

	Sleep(1000);

	return 0;

}

In theory our RTL should work on4096-waySGI* UV 1000 on Windows (http://www.sgi.com/products/servers/uv/specs.html). Are there anyvolunteersto check?:)

--Vladimir

What I think the task manager is failing to account for is the address space reserved by the threads as opposed to the page file space comitted. Let's see if I can explain (surmise) this.

When your test program starts, and runs up to, but before OpenMP starts, your virtual memory address space is something like this (order may differ

(4KB reserved) at 0x00000000
(static data) at +4KB
(code)
(initial heap)
(unmapped address) 2GB/3GB less above and below items
(reserved 4KB)
(main thread stack)
--------------------
0x80000000 or 0xC0000000 to 0xFFFFFFFF system address space of your virtual memory

If/when the heap expires prior to or following additional thread allocations, additional heaps are mapped/allocated/reserved from the unmapped address space (a portion thereof), assuming there is available address space.

Now then, when a new thread is allocated/created (the surmise part):

The O/S checks the unmapped address space to see if it has sufficient space for:

thread stack (default 1MB, you may specify differently)
guard page (4KB on x32)
optional thread context information (?KB)

These addresses come out of the virtual memory address space (assuming address space available)

*** Now then, until something is pushed onto the thread stack, more specifically a thread stack page (4KB page granularity), that formerly was an untouched page (4KB) of the thread's stack, had a reservaton of 4KB of the virtual address, but until touched, did not require physical memory nor page file space. The attempted touch causes (would cause) a page fault, then the O/S would map the page (assuming available page file space). A similar thing happens each time you add an additional heap (expand the heap).

What this means is your 981 threads have:

981x (default thread stack + 4KB guard) virtual address space consumed (~1GB)
981x (4KB touched stack + 4KB guard) RAM/pagefile space consumed (~8MB)

When the program attempts to allocate the 982nd thread there is no available virtual address space.

At least this is my assessment as to what you are observing.

As TimP ponted out, in OpenMP, creating more threads than you have logical processors is generally counter-productive.

Jim Dempsey

>>A maximum number of threads I was able to see was18,623!

Yes, you can (by reducing the stack size) but on x32 what is the point?

In a compute bound system, more software threads than available hardware threads, is generally counterproductive. There may be a few outlier cases where a bad algorithm may see better performance (I should say may work). An example might be a poorly written mesh filter where node progress is blocked by waiting for other node(s) to complete. A better way to write this type of program would be to use a tasking based system where the software thread migrates from task to task as opposed to having more threads.

Jim Dempsey

I guess that some people get a knock out of testing whether compiler conforms to pubished specifications :)

Good for you Sergey, I'm wondering whether you can find apractical application for your expiriments. --Vladimir