Yet another stack overflow...

Yet another stack overflow...

Hello people!

I am using Debian Linux with te ifort version 10.1 and am developing a procedure for scientific purposes that explores a cluster of elements on some sort of a mathematical graph and extracts some information about that cluster. The procedure is recursive and it works fine up till some limiting size of the cluster (about 500000 elements in the cluster). When the size is exceeded the stack overflows and the program breaks. I am aware that it has to break at some point and I am fine with that, but what I do not understand is the fact that I can not postpone this ocurrence. What it says is:

"forrtl: severe (174): SIGSEGV, segmentation fault occurred"

I first I have reduced considerably the variables in the recusrsive procedure and it did not help at all, secondly I have tried to send instructions to "ld" to increase stack size by including a switch "-Xlinker --stack xxx" in the ifort command and it said "ld: unrecognized option '--stack'" which is curious since that switch can be found in the man pages of ld. The third thing I tried is using "ulimit -s unlimited" command and hence increased the stack size within bash - still no help to my problem. And one more thing, I am pretty sure that none of my integer variables exceed their max values.

The curious thing is that the program uses the same amount of ram in all the cases which is much less (150mb) than the amount of ram on my computer at the time it breaks. I wonder why does he not allocate the additional ram if he needs it? I was wondering if somebody has some insight in the way the recursive subroutines compile with ifort and has a solution of some sort. Remember, I do not need magic, I just want it to break because it has no more ram available if it has to break because of something :). I would rather not have to decompose the thing in do loops if I can somehow increase the available memory by manipulating ifort and/or linker commands.

Thanks very much in advance, Ivan Balog.

7 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Ivan,

Is this a single threaded app or OpenMP app?

How many levels of recursion is taking place before the error?

If you write out the locaton of a stack local variable how much stack is consumed on each recursion?

If your recursion is passing a fixed content + variable content list of arguments as it nests deeper can you make the fixed content a comon type and then pass a single referenct to that type?

call foo(A,B,C,D,E,F,G,i,j)

recursive subroutine foo(A,B,C,D,E,F,G,i,j)
REAL :: A,B,C,D,E,F,G
INTEGER :: i,j
...
if(i .LE. iMax)call foo(A,B,C,D,E,F,G,i+1,j+1)
...
end subroutine foo

----------- change to ------------------

type FooContext
REAL :: A,B,C,D,E,F,G
end type FooContext

...
type(FooContext) :: context
...
context%A = A
...
context%G = G
call foo(context,i,j)

recursive subroutine foo(context,i,j)
type(FooContext) :: context
...
if(i .LE. iMax)call foo(context,i+1,j+1)
...
end subroutine foo

---------------------------------------

Jim Dempsey

Dear Sir,

It is a simple single treaded program.

Thank you very much for your swift reply. Can I understand from your post that if i bundle up all variables I can in a single type that it will take less stack? That is a very useful remark.

It appears that the recursion blows up at the depth of 400000 to 500000, long before any of the 8 byte integer variables have a possibility to "turn around". This indicates a stack problem and and I want to compile the program in a way for it to ask for more stack if it needs it. How do I do that?

A recursion of 500,000 levels is quite large.

What did you observe for the stack consumed per level?
(taking the difference between the LOC() of a stack local variable)

It would be nice to know what size we are looking at.

Jim

Quoting - jimdempseyatthecove

A recursion of 500,000 levels is quite large.

What did you observe for the stack consumed per level?
(taking the difference between the LOC() of a stack local variable)

It would be nice to know what size we are looking at.

Jim

I am aware that this is a large recursion, and this is not really a program for general use so to say, it is designed to do a very specific task and I do not care if it takes up all of the ram on the machine it is executed on.

I do not really know how to see exactly how much stack it uses. I can estimate it as follows: my local variables take 36bytes at each call and to this you need to ad a pointer which says to the procedure where to come back. That is no more than 50bytes. If you multiply this by the depth of 500000 you get an amount of no more than 25mb of stack which I never specified and yet they exist as a hard limit!! I am aware of the fact that I could have written the thing by now in the loops and solved the problem. But I simply want to know how do I tell the compiler to either use unlimited amount of stack or to specify the limited amount myself which is better than this - not knowing the limit.

Ivan.

Best Reply

Ivan,

You are estimating a value for stack consumption not measuring the value. Measuring the value will take less than 1 minute (break, look at esp/rsp, after recurse, break, look at esp/rsp, subtract, produce difference).

If the stack consumption is on the order of 50:100 bytes then there is nothing unusual (excepting your estimate is off by 1/2).

However, if the stack consumption (per level)is substantially larger then something else is going on that is not obvious to you. Resolving this will help you understand the problem, understanding the problem will help you devise a work around (solution).

Your recursive subroutine/function may (unbeknownst to you) be creating stack temporaries of substantial size (e.g. a temporary array).

If this is a case of temporary arrays consuming space then the statements that produce the temporary arrays may be able to be re-written or exported to a small subroutine (that allocates the temporary and deallocates the temporary array).

Jim Dempsey

Quoting - jimdempseyatthecove

Ivan,

You are estimating a value for stack consumption not measuring the value. Measuring the value will take less than 1 minute (break, look at esp/rsp, after recurse, break, look at esp/rsp, subtract, produce difference).

If the stack consumption is on the order of 50:100 bytes then there is nothing unusual (excepting your estimate is off by 1/2).

However, if the stack consumption (per level)is substantially larger then something else is going on that is not obvious to you. Resolving this will help you understand the problem, understanding the problem will help you devise a work around (solution).

Your recursive subroutine/function may (unbeknownst to you) be creating stack temporaries of substantial size (e.g. a temporary array).

If this is a case of temporary arrays consuming space then the statements that produce the temporary arrays may be able to be re-written or exported to a small subroutine (that allocates the temporary and deallocates the temporary array).

Jim Dempsey

Thank you again very much for your persistent help! I observed as you said the amount of stack taken by each recursion call. It was 54bytes to be precise. The solution to the problem was indeed in the limited stacksize but I tought that the stack size is determined by the compiler which is very wrong - it is determined in bash (somewhat unintuitive to me at least). I managed to change my maximum allowed stack size by including the following line in the "/etc/security/limits.conf" (via root privileges)

user hard stack #value#

and subsequently adding a line

ulimit -s #value#

to the .bashrc of my user's home.

After these changes the program works just as I want it to - If it needs more stack it simply allocates some more.

Leave a Comment

Please sign in to add a comment. Not a member? Join today