Can you show the code?

From your description, the amount of work that SUB has to do depends on how long it takes OPER to finish ("SUB has a DO WHILE loop that goes until OPER tells it to quit").  In the serial case, doesn't that mean SUB does nothing - OPER has already finished?  If so, that means the two cases are far from equivalent. 

A spinning DO WHILE loop will tie up a core (in the absence of the compiler working out that the loop does nothing and eliminating it).  If you put a sleep inside the loop then the core becomes available for work by other threads in the system - in the case where you have less cores than active system threads (how many cores do you have?) that could affect the number of timeslices given to the thread running OPER.

Perhaps I've misunderstood, but if not consider giving SUB a fixed unit of work.

Making a shared variable volatile is not on its own enough to avoid data race conditions and/or guarantee a consistent view of the variable between threads.  Based on your description I expect you would need explicit synchronisation and flush operations.  How you have those arranged can also make a significant difference to execution time.

The sleep response could be explained by your threads sharing the same physical core.  I think default thread affinity depends on what the operating system had for breakfast.  Out of my domain, others will know better. 

In the meantime open a command prompt, then:

set KMP_AFFINITY=verbose,scatter

then run your program in that command prompt and see what happens.  This should force threads to different physical cores and give you some diagnostics as well.

This may be completely unrelated to your problems,but if by "built in flags" you mean ordinary Fortran variables in conjunction with ordinary Fortran statements, then it is probable that your synchronisation is not formally well defined.

>>But my goal is to bufffer data in the OPER that is written out in SUB.

Do you intend to run your code with OpenMP Nested enabled?
(i.e. does OPER contain !$OMP PARALLEL...?)

Is OPI called from within a parallel region?
Is OPI called only once or many times?
Is the results data produced by OPER large or small?
What will the ratio be of time spent by SUB verses time spent by OPER?

The concept in the mentioned link is for a technique to overlap writes (or reads) with work.
And this recommended only when the writes (or reads) are a significant portion of the work.

The construction of your code in this forum thread is non-overlapped (no advantage to parallelization).
Not knowing about your code it is difficult to recommend a technique.

Presumably OPER has a loop. If so, can the partial results be written (SUB) as they are accumulated (iow on/after each iteration)?
If OPER has a loop, then you have at least two differen strategies:
a) single buffer (one results buffer, one intermediary copy for writing)
b) double buffer (two results buffer, no intermediary copy for writing)

Method a) is often easier to implement but has the overhead of copying data in memory.

Additional information will be required before we can make recommendations.

Jim Dempsey

The OpenMP spec says "...if at least one thread reads from a memory unit and at least one thread writes without synchronization to that same memory unit ... then a data race occurs.  If a data race occurs then the result of the program is unspecified."  Beyond that, there's the need to ensure that your threads view of any shared variables is consistent.

I still think that the simple explanation is that your spinning DO WHILE loop is not equivalent to your serial case.  Again, out of my domain in terms of what happens at a hardware level, but I could imagine that the continuous access of a shared flag variable in the loop introduces reasonable overhead.

Discussion of this sort of topic is difficult without specific code examples (e.g. depending on the number of threads you start inside OPER you can be back in the situation of having less physical cores than threads wanting to run).  Based on what I think you are trying to do I've attached an example that uses omp locks. I think this is formally correct, but note that I really only dabble in OpenMP - I can usually get myself into enough trouble when running things serially that I only contemplate running in parallel when I wish for almost certain disaster.  For this specific example (noting that there are ten batches, sub and oper take about the same time per batch and the output from the batches are somewhat distant from each other in memory) the parallel case leaves the serial case choking in its dust.  Vary things and you can elininate or even slightly reverse that relative performance.

OMP tasks might also be suitable for this - I've used them successfully in an application which also has producers and consumers of data, but with more complicated dependencies between actors.

!$ is the OpenMP free form source conditional compilation sentinel - see 2.2.2 of the OpenMP 3.1 spec.  Those lines are comments if OpenMP is not enabled, they are normal statments if OpenMP is enabled.  You shouldn't need to edit the source - it should (hopefully) compile and behave similarly regardless of OpenMP.

No, but my hardware is nowhere near as capable as yours.  Make the number of iterations (count) bigger, perhaps by a factor of 100, then secondly perhaps set the affinity to scattered and see what happens.

If only one thread is doing work, as appears to be implied, OpenMP could be expected to slow it down.  Reducing the value of KMP_BLOCKTIME might make a difference, as you are discussing times of that order of magniitude.

It might help of you would state what you are trying to learn from this thread; but you've already declined to answer questions leading in that direction.

In the example you posted, doesn't OPER get executed by both threads (watch out for races), followed by SUB being executed by one thread, followed by a wait for KMP_BLOCKTIME timeout?

Your understanding (that there is an implicit section directive after the parallel sections directive) is the same as mine.

When I tried my example code (with larger iteration counts) on a four physical core machine with hyperthreads (8 logical cores) Windows 7 machine I got similar variable/unexpected results to you.  On my ancient two physical core no hyperthreading Vista machine I see consistent speedup with the OMP case.

The call to random_number might be (i.e. I don't know) invoking additional synchronisation in my example that I didn't count on.  More tomorrow.

IanH,

I took the liberty to modify your program. I haven't tried all permutations of compile options.

Change summary:

Changed name of "sub" to "IOsub" to reflect purpose of subroutine.

Made Master thread call IOsub, (all) other thread(s) call OPER.

Outer level runs with all threads (you can set this to 2 threads if you want to enable nested.

Locks are set for region of work by each OPER worker thread that will perform work in OPER

Moved RANDOM_NUMBER out of inner loop (both in IOSUB and OPER)

Added nThreads = omp_get_num_threads(); iThread = omp_get_thread_num()

The DO ib loop in OPER now iterates

DO ib=iThread, batches, nThreads-1

When more than 2 threads in use then each non-master thread takes interlieved steps over batches.

Note, modifications assume all computation results maintained by application, IOsub outputs each batch as it is completed. The real application may not wish to hold all results (batches), in that event, double buffering or n-buffering could be considered.

Jim Dempsey

I forgot to mention.

Should you want to use nested parallelism, then set the number of threads on the outer level to 2. For test purposes then modify OPER DO i=1,count loop to have !$OMP PARALLEL DO SHARED(ib,count,r,rnd)

Jim Dempsey

On my system (Core i7 4-core w/HT), with IOsubs at count/4 and no thread limitation on outer leve (iow 8 threads).

 IOsubs at count/4
OpenMP stubs (sequential code) 1.29s
OpenMP parallel code .44s

IOsub at count/100
OpenMP stubs (sequential code) 1.085s
OpenMP parallel code .285s

Jim Dempsey

Have you tried running this under Intel VTune Amplifier XE to look for thread and lock contention, etc.?

dajum,

Programming error. You do not want me to do all your work for you...
See fix attached.

Jim Dempsey

bh2 has got the "issue" with data races that I was talking about before - you have writes to `index` and `done` potentially going on in parallel to reads of `index` and `data` (the if statement).  For your program to be formally well defined you need to synchronise those.  Similarly there's a formal requirement to ensure that the "view" of the shared variables is consistent between threads. 

I qualify with "formally" because the compiler's generated code and your hardware may "happen" to achieve both of those aspects practically (I don't know - I don't play at that level - whenever the debugger throws up disassembly I have to go and have a good lie down).  Even if those requirements are being met practically, that will come at some cost in terms of execution speed.

My attempt at fixing those aspects (and then perhaps breaking other things) attached.  Measured over four runs, on my two core, no HT system the omp case is faster by about 15% in a release build.  On a four core HT system similar.  Disabling HT on the four core, the omp run is 40% faster than serial (serial runtime is about the same HT to no HT). (I'm seeing occasional significant variability in both serial and omp cases on the four core machine which may be due to background system activity, disk buffering, etc - perhaps its also possible its due to a programming error.)

dajum,

I will be out of my office today, will look further and at bh3.f90.

In my tests I ran: In parallel and with OpenMP stubs. I did not run compiled without OpenMP.

My system has one socket. I do not know if that is affecting the issue, it shouldn't.

Jim Dempsey

Put a break point in (but prior to loop) both OPER and SUB (IOsub). Run in parallel build, verify that you are a) running in two threads, b) both threads are in seperate cores/HW threads. b) should be observable by starting the performance monitor and watching the CPU utilization during run.

On the notebook and single E5405 all theads will share the LL Cache (L3), on the multi-socket maching you may be running in different sockets, in that case a memory intensive could run longer (however, this should be the case for the 1st iteration of your inner loop (DO i)

*** this assumes the RANDOM_NUMBER has been moved outside the DO i loop. If you have not moved this outside the loop then both (all) theads are passing through a critical section on each iteration of their respective DO i loop (as opposed to once per batch).

Jim Dempsey

Revised program.

Edited results

KMP_AFFINITY = verbose
OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7}
OMP: Info #156: KMP_AFFINITY: 8 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 4 cores/pkg x 2 threads/core (4 total cores)
OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1,2,3,4,5,6,7}
Number of threads            1
Elapsed time: 18.56 s
Number of threads            2
OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1,2,3,4,5,6,7}
Elapsed time: 19.93 s
Number of threads            3
OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {0,1,2,3,4,5,6,7}
Elapsed time: 10.55 s
Number of threads            4
OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {0,1,2,3,4,5,6,7}
Elapsed time: 8.439 s
Number of threads            5
OMP: Info #147: KMP_AFFINITY: Internal thread 4 bound to OS proc set {0,1,2,3,4,5,6,7}
Elapsed time: 6.533 s
Number of threads            6
OMP: Info #147: KMP_AFFINITY: Internal thread 5 bound to OS proc set {0,1,2,3,4,5,6,7}
Elapsed time: 4.765 s
Number of threads            7
OMP: Info #147: KMP_AFFINITY: Internal thread 6 bound to OS proc set {0,1,2,3,4,5,6,7}
Elapsed time: 4.720 s
Number of threads            8
OMP: Info #147: KMP_AFFINITY: Internal thread 7 bound to OS proc set {0,1,2,3,4,5,6,7}
Elapsed time: 4.720 s
Done
KMP_AFFINITY=verbose,compact
OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7}
OMP: Info #156: KMP_AFFINITY: 8 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 4 cores/pkg x 2 threads/core (4 total cores)
OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1
OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine
OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1}
Number of threads            1
lapsed time: 18.60 s
Number of threads            2
OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1}
Elapsed time: 20.33 s
Number of threads            3
OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3}
Elapsed time: 10.14 s
Number of threads            4
OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3}
Elapsed time: 12.07 s
Number of threads            5
OMP: Info #147: KMP_AFFINITY: Internal thread 4 bound to OS proc set {4,5}
Elapsed time: 7.059 s
Number of threads            6
OMP: Info #147: KMP_AFFINITY: Internal thread 5 bound to OS proc set {4,5}
Elapsed time: 5.014 s
Number of threads            7
OMP: Info #147: KMP_AFFINITY: Internal thread 6 bound to OS proc set {6,7}
Elapsed time: 5.022 s
Number of threads            8
OMP: Info #147: KMP_AFFINITY: Internal thread 7 bound to OS proc set {6,7}
Elapsed time: 4.995 s
Done
KMP_AFFINITY=verbose,scatter
OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7}
OMP: Info #156: KMP_AFFINITY: 8 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 4 cores/pkg x 2 threads/core (4 total cores)
OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1
OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine
OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1}
Number of threads            1
Elapsed time: 18.66 s
Number of threads            2
OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2,3}
Elapsed time: 19.79 s
Number of threads            3
OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4,5}
Elapsed time: 9.924 s
Number of threads            4
OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6,7}
Elapsed time: 8.047 s
Number of threads            5
OMP: Info #147: KMP_AFFINITY: Internal thread 4 bound to OS proc set {0,1}
Elapsed time: 6.172 s
Number of threads            6
OMP: Info #147: KMP_AFFINITY: Internal thread 5 bound to OS proc set {2,3}
Elapsed time: 5.061 s
OMP: Info #147: KMP_AFFINITY: Internal thread 6 bound to OS proc set {4,5}
Number of threads            7
Elapsed time: 4.853 s
Number of threads            8
OMP: Info #147: KMP_AFFINITY: Internal thread 7 bound to OS proc set {6,7}
Elapsed time: 4.704 s
Done
KMP_AFFINITY=verbose,compact,1,0
OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7}
OMP: Info #156: KMP_AFFINITY: 8 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 4 cores/pkg x 2 threads/core (4 total cores)
OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1
OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine
OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1}
Number of threads            1
Elapsed time: 18.62 s
Number of threads            2
OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2,3}
Elapsed time: 19.93 s
OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4,5}
Number of threads            3
Elapsed time: 9.921 s
OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6,7}
Number of threads            4
Elapsed time: 8.193 s
Number of threads            5
OMP: Info #147: KMP_AFFINITY: Internal thread 4 bound to OS proc set {0,1}
Elapsed time: 6.046 s
Number of threads            6
OMP: Info #147: KMP_AFFINITY: Internal thread 5 bound to OS proc set {2,3}
Elapsed time: 5.065 s
OMP: Info #147: KMP_AFFINITY: Internal thread 6 bound to OS proc set {4,5}
Number of threads            7
Elapsed time: 4.944 s
Number of threads            8
OMP: Info #147: KMP_AFFINITY: Internal thread 7 bound to OS proc set {6,7}
Elapsed time: 4.654 s
Done

Jim Dempsey

dajum,

The 2 thread issue (slower than 1) may be due to the code used to emulate the load for the IOsub. I suggest you modify this to perform a formatted internal write (to character variable), then call SLEEPQQ to emulate write latency. This may be more representative of your overhead for IOsub.

Jim Dempsey