First, read this article, which is listed at the top of this forum: http://software.intel.com/en-us/articles/determining-root-cause-of-sigsegv-or-sigbus-errors/

Next, for clusters keep in mind that some batch systems will not propagate env settings like 'ulimit -s unlimited'. You may want to make that the default for the cluster in the system wide /etc/profile.

ron

With higher levels of optimization there is more in-lining and higher unrolling of loops. When the in-lining and unrolling includes code that generates(uses) stack based temporaries then stack consumption goes up (possibly way up in the case where array temporaries are created).

I suggest you experiment with using options to control the extent of IPO, then loop unrolling, then in-lining. Keep O3 on. Do this globally, if this works, then start removing the restrictions. Once you find the group (IPO, loop unroll, in-lining) then start working on groups of files within your projects. IOW search for offending source files.

Elimination of the creation of array temporaries may be in order. There is a diagnostic option that triggers a once only printout of where an array temporary is created in your code.

Jim Dempsey

May I clarify something please?

You said:

My flags are:

FFLAGS=-O1 -xHost -fp-model precise -heap-arrays 100000 -mcmodel=medium (pls notice the amount of stack

Are you saying that "-heap-arrays 100000" is setting your stack?

If so, I'm afraid there is a bit of confusion. The "-heap-arrays " switch tells the compiler to put temporary arrays bigger than "" into heap, rather than using the stack. Setting that number to be so large only puts arrays bigger than 100K on the heap. You could still have several temps created on the stack, each of which are smaller than 100K.

You should follow the suggestion to look at how many temporaries you are creating, and see if it's possible to limit those.

--Lorri

In fact, -heap-arrays 100000 will put only fixed size allocations beyond that size on heap. Chances are that all your big allocations are variable size and go on stack.
Global stack size is set by your shell command. It's entirely normal to set "unlimited" (as Ron mentioned), which actually is a reasonable system-dependent limit, which you would set up to happen each time you open a shell on each node of the cluster, in case it's not reasonable to ask your sysadmin to make it happen for everyone.

I am not surprised that -ipo would have limitations in a project large enough to require -mcmodel. You are probably better off relying on same-file in-lining for the most frequent calls.
Using processor-dependent option such as -xHost doesn't have a clear connection to any problems you mention here. On Westmere X56xx I have never found -xHost running as well as -xSSE4.1 or possibly -xSSE2, but you haven't indicated any reasons why you might see a problem associated with these choices. Differences between -O2 and -O1 should have much more impact on performance than the architecture option. There is nothing wrong with choosing a single architecture option which works for all your clusters.
Intel MPI has borrowed a number of features from MVAPICH, so it is not so much a competition as an effort to include as many useful facilities as possible.