OMP crash during program exit

OMP crash during program exit

On MacOS i'm getting a crash during program exit when using libiomp5.dylib from Composer XE 2013.1.119.
The crash does not occur when using

  • compiler and libiomp5 from 2013.0.088 or older versions
  • using libiomp5 from 2013.0.088 or older versions while still compiling with 2013.1.119 and linking aginst it's version of libiomp5

Here is the libiomp5 module info:

185 libiomp5.dylib - 0x111e48000  dyld Y Y /Users/jascha/turbulence/package/TurbulenceFD_C4D_v1-0_1038/TurbulenceFD/libs/libiomp5.dylib at 0x111e48000 (offset 0x111e48000)

and the stack trace:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000114a76f40
0x0000000111ebe612 in __kmp_cleanup_user_locks ()
(gdb) bt
#0  0x0000000111ebe612 in __kmp_cleanup_user_locks ()
#1  0x0000000111ece2ad in __kmp_internal_end ()
#2  0x0000000111eceb30 in __kmp_internal_end_dtor ()
#3  0x00007fff5fc0f15c in __dyld__ZN16ImageLoaderMachO13doTerminationERKN11ImageLoader11LinkContextE ()
#4  0x00007fff5fc03591 in __dyld__ZN4dyld14runTerminatorsEPv ()
#5  0x00007fff935a17c8 in __cxa_finalize ()
#6  0x00007fff935a1652 in exit ()
#7  0x00007fff9256ee5a in -[NSApplication terminate:] ()
#8  0x00007fff8dfad70d in -[NSObject performSelector:withObject:] ()
#9  0x00007fff9266b8ca in -[NSApplication sendAction:to:from:] ()
#10 0x00007fff927587ff in -[NSMenuItem _corePerformAction] ()
#11 0x00007fff92758536 in -[NSCarbonMenuImpl performActionWithHighlightingForItemAtIndex:] ()
#12 0x00007fff926d60d4 in -[NSMenu performKeyEquivalent:] ()
#13 0x00000001009cd3a6 in catch_exception_raise ()
#14 0x00007fff926d500e in -[NSView _performKeyEquivalent:conditionally:] ()
#15 0x00007fff926d5259 in -[NSControl _performKeyEquivalent:conditionally:] ()
#16 0x00007fff926d513c in -[NSView performKeyEquivalent:] ()
#17 0x00007fff926d500e in -[NSView _performKeyEquivalent:conditionally:] ()
#18 0x00007fff926d513c in -[NSView performKeyEquivalent:] ()
#19 0x00007fff926d500e in -[NSView _performKeyEquivalent:conditionally:] ()
#20 0x00007fff926d4f81 in -[NSWindow performKeyEquivalent:] ()
#21 0x00007fff926d4d7a in -[NSApplication _handleKeyEquivalent:] ()
#22 0x00007fff925cde86 in -[NSApplication sendEvent:] ()
#23 0x00007fff92564a0e in -[NSApplication run] ()
#24 0x00000001009d53d0 in catch_exception_raise ()
#25 0x0000000100002194 in ?? ()

11 Beiträge / 0 neu
Letzter Beitrag
Nähere Informationen zur Compiler-Optimierung finden Sie in unserem Optimierungshinweis.

>>__kmp_cleanup_user_locks ()

Are (were) any of the lock objects located on the stack of a thread (which may have departed or poped out of existance)?
Note, no crash in different version may indicate you were lucky.

Jim Dempsey

www.quickthreadprogramming.com

Thanks for your reply!

There are no explicit lock objects in the program, i.e. no omp_init_lock or omp_init_nest_lock calls, only #pragma omp critical sections. Replacing all of them with custom locks prevents the crash.
Why would a #pragma omp critical not clean up the underlying lock properly?

At the time of the program exit there are no existing OMP threads except for the sleeping pooled threads. These have a stack trace like this:

#0 0x00007fff93510bca in __psynch_cvwait ()
#1 0x00007fff935b2274 in _pthread_cond_wait ()
#2 0x0000000110e1c57e in __kmp_suspend ()
#3 0x0000000110def96c in __kmp_hyper_barrier_release ()
#4 0x0000000110df9852 in __kmp_fork_barrier ()
#5 0x0000000110df8362 in __kmp_launch_thread ()
#6 0x0000000110e17dee in __kmp_launch_worker ()
#7 0x00007fff935ae8bf in _pthread_start ()
#8 0x00007fff935b1b75 in thread_start ()

>>Replacing all of them with custom locks prevents the crash.
>>Why would a #pragma omp critical not clean up the underlying lock properly?

Seems strange.

Is it possible that you enter the critical section with one thread...
one or more other threads attempt to enter critical section
thread holding critical section exits or issues unhandled structured exception
normal exit from critical section (never reached by holding thread).

Jim Dempsey

www.quickthreadprogramming.com

Hello!

We need some more information for further investigation.

Please, provide the output from setting KMP_VERSION=1 and KMP_SETTINGS=1 environment variables.

Also you mentioned the crash occurs on MacOS. Does it mean the crash dos not occur on Linux/Windows or you just tried only Mac?

More information about application itself would be useful as well. Actually we need to know how the application uses OpenMP (constructs, API).

It would be really useful to get a reproducer.

>>...Replacing all of them with custom locks prevents the crash...

Hi everybody,

>>Is it possible that you enter the critical section with one thread...
>>one or more other threads attempt to enter critical section
>>thread holding critical section exits or issues unhandled structured exception
>>normal exit from critical section (never reached by holding thread).

I understand that the crash happens on a MacOS platform and a critical section is somehow involved. So, I simply would like to give an example how I do clean ups for a critical section on a Windows platform ( I know that it could be not applicable for MacOS... ). I hope that it could help:

CDataContainer::~CDataContainer( RTvoid )
{
...
while( m_cs.RecursionCount > 0 )
::LeaveCriticalSection( &m_cs );

::DeleteCriticalSection( &m_cs );

::SetLastError( _RTCOMPLETED_SUCCESSFULLY );
}

As you can see there is a 'while' loop with a call to a Win32 API function 'LeaveCriticalSection' in order to properly release the critical section completely (!).

Zitat:

Olga Malysheva (Intel) schrieb:

Please, provide the output from setting KMP_VERSION=1 and KMP_SETTINGS=1 environment variables.

Intel(R) OMP Copyright (C) 1997-2012, Intel Corporation. All Rights Reserved.
Intel(R) OMP version: 5.0.20120914
Intel(R) OMP library type: performance
Intel(R) OMP link type: dynamic
Intel(R) OMP build time: 2012-09-14 06:12:26 UTC
Intel(R) OMP build compiler: Intel C++ Compiler 12.1
Intel(R) OMP alternative compiler support: yes
Intel(R) OMP API version: 3.1 (201107)
Intel(R) OMP tracefile support: no
Intel(R) OMP dynamic error checking: no
Intel(R) OMP helper thread support: yes
Intel(R) OMP thread affinity support: no
Intel(R) OMP debugger support version: 1.1

User settings:

KMP_SETTINGS=1
KMP_VERSION=1

Effective settings:

KMP_ABORT_DELAY=0
KMP_ABORT_IF_NO_IRML=false
KMP_ALIGN_ALLOC=64
KMP_ALL_THREADPRIVATE=128
KMP_ALL_THREADS=2147483647
KMP_ASAT_DEC=1
KMP_ASAT_FAVOR=0
KMP_ASAT_INC=4
KMP_ASAT_INTERVAL=5
KMP_ASAT_TRIGGER=5000
KMP_ATOMIC_MODE=2
KMP_BLOCKTIME=200
KMP_DETERMINISTIC_REDUCTION=false
KMP_DUPLICATE_LIB_OK=false
KMP_FORCE_REDUCTION: value is not defined
KMP_FOREIGN_THREADS_THREADPRIVATE=true
KMP_FORKJOIN_BARRIER="2,2"
KMP_FORKJOIN_BARRIER_PATTERN="hyper,hyper"
KMP_GTID_MODE=0
KMP_HANDLE_SIGNALS=false
KMP_INIT_AT_FORK=true
KMP_INIT_WAIT=2048
KMP_ITT_PREPARE_DELAY=0
KMP_LIBRARY=throughput
KMP_LOCK_KIND=queuing
KMP_MALLOC_POOL_INCR=1M
KMP_MONITOR_STACKSIZE: value is not defined
KMP_NEXT_WAIT=1024
KMP_NUM_LOCKS_IN_BLOCK=1
KMP_PLAIN_BARRIER="2,2"
KMP_PLAIN_BARRIER_PATTERN="hyper,hyper"
KMP_REDUCTION_BARRIER="1,1"
KMP_REDUCTION_BARRIER_PATTERN="hyper,hyper"
KMP_SCHEDULE="static,balanced;guided,iterative"
KMP_SETTINGS=true
KMP_SSP_MAX_THREADS=0
KMP_STACKOFFSET=0
KMP_STACKSIZE=4M
KMP_STORAGE_MAP=false
KMP_TASKING=2
KMP_TASK_STEALING_CONSTRAINT=1
KMP_USE_IRML=false
KMP_VERSION=true
KMP_WARNINGS=true
OMP_DYNAMIC=false
OMP_MAX_ACTIVE_LEVELS=2147483647
OMP_NESTED=false
OMP_NUM_THREADS: value is not defined
OMP_WAIT_POLICY=PASSIVE

After program exit has begun, i get another line:

Intel(R) OMP Intel(R) RML support: not using

Zitat:

Olga Malysheva (Intel) schrieb:

Also you mentioned the crash occurs on MacOS. Does it mean the crash dos not occur on Linux/Windows or you just tried only Mac?


On Windows i don't get the crash. There's no Linux build of the application.

Zitat:

Olga Malysheva (Intel) schrieb:

More information about application itself would be useful as well. Actually we need to know how the application uses OpenMP (constructs, API).

It would be really useful to get a reproducer.


Unfortunately, i won't have the time to create an isolated reproducer.
I can guide you through setting up the software and making it crash, though.

Yes, please, give detailed instructions how to see the crash.
Thank you.

>>...Unfortunately, i won't have the time to create an isolated reproducer...

Please take into account that your case could be given a lowest priority and it possible that it won't be resolved for a long time. How an Intel software engineer could reproduce it? What would you personally do in a similar case?

Olga Malysheva (Intel) already asked for detailed instructions from you...

Quote:

Olga Malysheva (Intel) wrote:

Yes, please, give detailed instructions how to see the crash.
Thank you.


You should have received a message with files and detailed instructions.

Update: this opm crashing issue is fixed in the 13.0 update 3.

Thanks for reporting it.

Jennifer

Kommentar hinterlassen

Bitte anmelden, um einen Kommentar hinzuzufügen. Sie sind noch nicht Mitglied? Jetzt teilnehmen