Forum Jump

Select Group :
Select Forum :
Sorted By :
Sort Order :
From The :
 
Thread Tools  Search this thread 
Amit
Total Points:
420
Status Points:
370
Green Belt
July 2, 2009 5:14 PM PDT
OpenMP-MPI hybrid code - SIGSEGV
Hi,

I have a hybrid MPI-OpenMP fortran code which I run on 2 OpenMP threads and 1 MPI processor.
Everything works correctly. (The code is loop level parallelized.)

When I tried to run the code on 4 threads I got NANs in my solution.

I went through previous forum posts and found that if I include the flag -shared-intel -mcmodel=medium
then I can overcome the memory problem but now I run into following segmentation fault.
I also went through the segmentation error guide and couldn't find an answer to this problem.
(I have also tried setting the KMP_STACKSIZE to 25GB and ulimit -s unlimited)
btw, I use ifort 10.0.

amit@sys:~/openmp> mpirun -np 1 ./2009.x
MPI: On host sys, Program /openmp/2009.x, Rank 0, Process 26346 received signal SIGSEGV(11)


MPI: --------stack traceback-------
MPI: MPI_COMM_WORLD rank 0 has terminated without calling MPI_Finalize()
MPI: aborting job
MPI: Received signal 11

Thanks,
Amit
tim18
Total Points:
66,397
Status Points:
66,397
Black Belt
July 2, 2009 6:54 PM PDT
Rate
 
#1
Quoting - amit

I also went through the segmentation error guide and couldn't find an answer to this problem.
(I have also tried setting the KMP_STACKSIZE to 25GB and ulimit -s unlimited)
btw, I use ifort 10.0.

amit@sys:~/openmp> mpirun -np 1 ./2009.x
MPI: On host sys, Program /openmp/2009.x, Rank 0, Process 26346 received signal SIGSEGV(11)

You could try the -check and -g options to find out where you have problems. I wouldn't expect to find answers without following up Ron's suggestions about how to dig deeper. In principle, thread checker should be applicable, although it hasn't had much maintenance lately.
Why are you using such an old compiler?
When running 1 mpi process, it shouldn't matter which MPI you are using, but if your MPI is as old as your compiler, you are probably wasting your time.     The current Intel and HP MPI have built-in support for hybrid.
Ronald W. Green (Intel)
Total Points:
11,687
Status Points:
11,187
Brown Belt
July 3, 2009 7:24 AM PDT
Rate
 
#2 Reply to #1
Quoting - tim18
You could try the -check and -g options to find out where you have problems. I wouldn't expect to find answers without following up Ron's suggestions about how to dig deeper. In principle, thread checker should be applicable, although it hasn't had much maintenance lately.
Why are you using such an old compiler?
When running 1 mpi process, it shouldn't matter which MPI you are using, but if your MPI is as old as your compiler, you are probably wasting your time.     The current Intel and HP MPI have built-in support for hybrid.

As Tim says, do try -g -traceback and see if anything comes from that.

In older MPI implementations I have seen errors where environment is not propagated to child processes spawned by mpirun.  I would write a simple script:

#!/bin/bash
env

and run this under mpirun:  mpirun -np 1 ./myscript

just to see if your stacksize and KMP variables are propagating.  You could/should add those ulimit and KMP definitions to your login scripts   .profile  .cshrc or whatever ones you use.


Intel Software Network Forums Statistics

8285 users have contributed to 31229 threads and 99107 posts to date.
In the past 24 hours, we have 9 new thread(s) 39 new posts(s), and 55 new user(s).

In the past 3 days, the most popular thread for everyone has been comparison cilk++, openmp, pthreads first results The most posts were made to comparison cilk++, openmp, pthreads first results The post with the most views is Very amusing...  Escalated as

Please welcome our newest member tvinni