Problems with OpenMP; fftw (long, sorry)

Problems with OpenMP; fftw (long, sorry)

Hi all,

I've recently created a Makefile for building an image processing package with ifc. Previously on the linux platform pgi was the supported compiler. The code is f90. Thanks to the forum for help with a couple of issues along the way. Now that the code compiles and links, I am having some runtime errors with OpenMP and my use of an external library, fftw (fastest fourier transform in the west). The platform is Red Hat 8.0, ifc is version 7.0 (see comments below about compatibility).

Problem 1: I compile with -openmp and explicitly link -lguide, -lpthread, and -lPEPCF90. I do not use -parallel. During compile I get many messages that OpenMP routines are being parallelized. Yet when I run the code, only one CPU is being used. The machine is a dual-xeon with hyperthreading kernel enabled, and OMP_NUM_THREADS is set to 2 (have also tried 4), although this should not be necessary.

Problem 2: I've compiled the thread version of fftw, and even compiled the fortran interface portion of fftw with ifc. I link the fftw libraries successfully in my executable. But when code is reached that uses fftw, the program either segfaults (+core, if more than 1 thread was requested) or silently dies (no core, if only 1 thread was requested). I've looked at the corefile with gdb, stacktrace seems to show the crash occur after a couple of libpthread calls. The pthread lib linked (dynamically) here is /lib/i686/libpthread.so.0. There is a different version of libpthread.so in /lib (same version 0.10, but different filesize).

Recalling concerns with compatibility of ifc 7.0 and the glibc 2.2.93 included with Red Hat 8.0 (although I've not previously had problems with other programs I've built under this combination), I tried using ifc 6.0 with my code. This gives me an internal compiler error during compile, about 75% of the way through.

From searching this forum, I also see references to the need for libpthread to be built with support for FLOATING_STACKS. I don't know how to check if this is the case in Red Hat 8.0.

Other misc. compiler flags I'm using: -O2 -WB -mp -fpp2 -auto -pc64 -c -cm -w95 -tpp7 -xW

Thanks in advance for any suggestions, regarding either of these issues.

12 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Perhaps the following item from the release notes is of help - this is not a topic I am familiar with. But I can tell you that RH8.0 is not supported by Intel Fortran 7.0.

Steve

OpenMP Limitations

POSIX threaded programs that require a large stack size may not run correctly on some versions of Linux because of hard-coded stack size limits in some versions of the Linux POSIX threads libraries. These limits also apply to OpenMP programs (-openmp) and automatically generated parallel programs (-parallel) with the Intel compilers, because the Intel compilers use the POSIX threads library to implement OpenMP based and automatically generated parallelism. Threaded programs that exceed the stack space limit usually experience segmentation violations or addressing errors.

To avoid these limitations, please use a version of glibc built with the FLOATING_STACKS parameter defined, typically version 2.2.4 or later for both IA-32 and Itanium Processor Family. Then use the ulimit -s .... command to set the maximum shell stack size to an explicit large value (units of KBytes) and also set the KMP_STACKSIZE environment variable to the needed thread stacksize in bytes. A shell stacksize limit of unlimited does not work - it causes a fixed hard limit to be imposed. Note, in the bash shell, ulimit -s can be used to set a large maximum stack size only once. In the C shell (csh), ulimit -stacksize can be used to reset the maximum stacksize repeatedly. The default values for KMP_STACKSIZE have been increased to 2 MB for IA-32 and 4 MB for Itanium-based systems.

This solution has been tested on glibc version 2.2.4-13 for IA-32 and glibc 2.2.4-19 for Itanium Processor Family as found in the RedHat 7.2 Linux distribution. For glibc 2.2.4-13 on IA-32, the shared version of the POSIX threads library must be used, (there should not be a -static flag in the compiler .cfg file or on the command line).

In addition, if a common block is declared as "THREADPRIVATE" with an OpenMP directive, the common block must have the same length in all the source files in which it is declared.

Steve - Intel Developer Support

I would add that any libraries you are linking in need to be built with the Intel compilers. The lone exception is if the library was implemented in C, in which case gcc should work ok. However, linking in a library built by pgi90 or g77 isn't going to fly.

Also, the 6.0 compiler doesn't support Red Hat* 8.0 either, so while it shouldn't internal error, I wouldn't expect it to be any more effective than 7.0.

Brandon
Intel Developer Support

For on-line assistance: http://support.intel.com/support/performancetools
For product support information: http://www.intel.com/software/products/support
* Intel and Pentium are registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries
* Other names and brands may be claimed as the property of others

Brandon Hewitt Technical Consulting Engineer For 1:1 technical support: http://premier.intel.com Software Product Support info: http://www.intel.com/software/support

For completeness, I understand that in Red Hat 8.0, the static pthreads library is built without FLOATING_STACKS, and the shared pthreads library is built with FLOATING_STACKS. Nevertheless, there is no clear evidence that this is the problem. There are certainly additional issues of consistency that arise when an OpenMP program is threaded at two different levels.

An updated compiler with support for Red Hat 8.0 is expected in the fairly near future. That should eliminate one of the uncertainties.

Martyn

Thanks, all, for the replies!

@blhewitt:

> I would add that any libraries you are linking in need
> to be built with the Intel compilers.

Yes, I had linking problems (symbols not found) until I compiled the fortran wrapper portion of FFTW with ifc instead of g77. But I think that is the only external library I'm linking that is not either C (libc, libpthread) or intel-supplied (libPEPCF90, libguide).

> Also, the 6.0 compiler doesn't support Red Hat* 8.0
> either, so while it shouldn't internal error, I
> wouldn't expect it to be any more effective than 7.0.

Alas. I guess I will have to set up a development box with RH 7.3. It's funny, because I recently upgraded all of my machines to 8.0 partly because 7.3 comes with a bad gcc interim release that caused us many problems. So, I will need one box for C development & another for fortran :)

@Martyn:

> I understand that in Red Hat 8.0, the static pthreads
> library is built without FLOATING_STACKS, and the
> shared pthreads library is built with FLOATING_STACKS.

Thanks, that is good to know. But I am linking dynamically to libpthread, so apparently this is not my problem.

There are actually three unique different versions of libpthread on my RH 8.0 machine:

/usr/lib/libpthread.a
/lib/i686/libpthread-0.10.so
/lib/libpthread-0.10.so

The two .so versions are unique. I'm currently linking to the one in /lib/i686.

> There are certainly additional issues of consistency
> that arise when an OpenMP program is threaded at two
> different levels.

Sorry, could clarify your thoughts here?

> An updated compiler with support for Red Hat 8.0 is
> expected in the fairly near future.

Good news, hopefully within a couple of months?

In case it is of use, here is a typical corefile backtrace for one of my "Address error" segfaults when attempting to use OpenMP. Ah, libpthread & libc...

(gdb) bt
#0 0x083230b1 in locatenextlnb ()
#1 0x08322f0e in s_ndiag ()
#2 0x08322e29 in f_onsignal ()
#3 0x400ce47e in __pthread_sighandler () from /lib/i686/libpthread.so.0
#4
#5 0x42074033 in _int_free () from /lib/i686/libc.so.6
#6 0x42074a2c in free () from /lib/i686/libc.so.6
#7 0x400cb398 in __pthread_destroy_specifics () from /lib/i686/libpthread.so.0
#8 0x400c765a in __pthread_do_exit () from /lib/i686/libpthread.so.0
#9 0x400c894d in pthread_start_thread () from /lib/i686/libpthread.so.0

thanks,
jfk.

I decided to test whether my problems were solely due to Red Hat 8.0 incompatibility with ifc 6/7. So I installed Red Hat 7.3 on another machine & set up the development environment there. The results are interesting.

1 - ) My OpenMP threads work on the RH 7.3 box, after recompiling my application there. Progress!

2 - ) OpenMP threads do not work when this RH 7.3-built version is moved to a RH 8 machine.

3 - ) When I move a RH 8 built version over to the RH 7.3 machine, now MP threads do work!

So this seems to be a runtime problem, only. But, I've even tried statically linking everything including glibc and libpthread, & compiling under RH 7.3. But as soon as this build is run on a RH 8 machine, MP threads don't work.

It would appear that ifc (v. 7 here) can't generate MP code that will run under RH 8, regardless of what platform it's compiled under. Is this a known issue, or is there something I've not tried?

Um, you've already been told that ifc 7 doesn't support RH8... Linux has a long and sordid history of breaking binary compatibility across releases, and with RH8, the saga continues...

Steve

Steve - Intel Developer Support

Hi,

Thanks for posting this interesting issue. I would suggest running ldd to see what libs are dynamic & shared. If the libraries are all statically linked, it might be related to changes in pthread/glibc/kernel in RH 8.0 which is causing the problem (2). As Steve mentioned, ifc 7.0 does not support Red Hat 8.0, and as Martyn said, we will having an update shortly that will support the newer versions of glibc/kernel. We expect this to be in several weeks.

As Steve posted earlier in this thread (from the release notes), the compiler generates multi-threaded applications using the pthreads libraries. So it's either a problem with eithier correctly generating the pthread calls (doesn't seem as likely because it correctly works under the supported glibc/kernel versions) or that pthread/glibc/kernel changed in the OS. Do you have a small test case that illustrates the problem ?

regards,
John O'Neill

@Steve: Apologies if I was being redundant. I assumed the ifc/RH8 incompatibility was as a development platform, not deployment platform.

@John: I've tried both dynamically & statically linking the suspected libraries (glibc & pthread). Same result each way. After seeing how easy it is to build & run on Red Hat 7.3 I think I'm just going to wait this one out and continue using pgf90 until the new intel version is released. A few weeks isn't too long to wait.

A maybe interesting coincidence I've just noticed is that with their 4.0-2 release, Portland Group is no longer supplying their own pgthread library for posix threads. They seem to believe the problems in Red Hat's libpthread have been "fixed" in RH8.0. (I believe the issues in question related to the same FLOATING_STACKS support that has been discussed in this forum.)

Thanks very much for your replies & information.

cheers,
jfk.

I'm trying to compile some programs I know that work correctly with this compiler and they get compiled without problem, but when I try to execute them I always get this message:

abort: stack overlapping detected!
Perhaps you are using over 2 MB of stack with unsupported pthreads library that was built without FLOATING STACKS defined? This is the default for the static pthreads library and many unsupported operating systems.

I'm working with:

Operating System: Red Hat Linux 9
kernel: 2.4.20.8-smp
glibc: 2.3.2-11.9

Where can be the problem?
How can I solve it?

Thanks a lot

ALBERTO

On the same topic, two followup questions --

The first is perhaps foolish, but I don't know the answer. How difficult is it to simple build another glibc version (ie, 2.2.4 since it is compatible with ifc) to use for development, on a box that uses a later glibc as its system C library? I don't wish to replace this, just to have another available & statically link to it instead.

The second regards static linking. I can't seem to get my program to statically link my fftw libraries. I have the archive versions of the libs in the same place as the shared versions, which are currently being dynamically linked. I provide this directory in the makefile LINKLIBS section with -L. And I use the -Bstatic fflag for ifc. Yet the libraries always are dynamically linked.

What am I doing wrong here?

thanks,
jfk.

Okay, I solved my static linking problems. The rest of my questions are still outstanding, though.

Leave a Comment

Please sign in to add a comment. Not a member? Join today