Stack Overflow and setting Stack Size

Stack Overflow and setting Stack Size

I converted FORTRAN 77 code into dynamically allocating arrays. The size of the arrays are based on counting from a file that is read by one of the subroutines. I've discovered I need to set the stack size at 2MB for a 6MB read file, 6 MB for a 31 MB read file and I'm now at a stack size of 17MB for a 90 MB read file. These read files could be several GB in size. Is there ANY way to avoid having to reset the stack size all the time? I use EDITBIN rather than recompiling all the time. I can easily reset the stack size (but I'm a programmer and have the compiler), whereas our users won't be able to do this and we want to make this all invisible to them and just let them run any size of read files (which is why we went to dynamical allocation in the first place). Any answers out there????

19 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

There's got to be more to the story if you are indeed dynamically allocating arrays. How about sending a short example to us at vf-support@compaq.com and we'll take a look.

Steve

Retired 12/31/2016

Steve,
I've been working all day trying to replicate the stack overflow run-time error on a small set of program, but I haven't been able to yet. The system I'm working on has 317 programs, one comon, and one module. There are 578 allocatable arrays, and there is alot of processing going on, so maybe I won't be able to replicate it. In trying to replicate the run-time error, I created 20 integer, 4 character and 2 real allocatable arrays and it all work beautifully. I'll keep trying. All I know is, with our system I get that darn stack overflow, then I use EDITBIN to increase the size i.e. EDITBIN /swap:17000000 .exe (for a 17MB stack) and then it works fine. Why can't the compiler use the default stack (1MB) and then increase the stack on its own dynamically, based on the size of the allocatable arrays? Back to replicating the problem. Thank you so much for responding.

Unlike on more sophisticated operating systems, the stack size on Windows is fixed - there's nothing the compiler can do to increase it. I suspect the problem comes when the compiled code decides it needs to make a temporary copy of one of your arrays - if it does so, it does so on the stack. (We have work in progress to allow for heap allocation of temps in a future version.)

Usually, it's possible (and even desireable) to restructure the code to avoid the use of temps. Have you tried 6.5A or 6.6 - 6.5A eliminated many unnecessary array copies.

The thing to do is to find out what statement is executing when the error occurs, and look at it to see why a stack temp is required. We can help with that part if you can send us the routine in question (along with any code needed to compile it) and an indication of where in the routine the error occurs.

Steve

Retired 12/31/2016

Steve,
No, I haven't tried v6.5a yet. I'll download it tomorrow and try that. The exact place the compiler gives the stack overflow error is the first line of a subroutine (the one that happens to use alot of the very huge arrays), which says:
SUBROUTINE CONSTR2 (with 2 char variables and 2 logicals passed in--no arrays because they're in COMMON).
Actually, CONSTR2.F is 3,000 lines long and I've thought about making it into 2 or 3 subroutines rather than one long one. Do you think the size of the subroutine itself could be causing problems? All other subroutines are 1,000 lines or less. Thanks you for your help!

Look for array declarations in CONSTR2 with bounds dependent on a variable (in COMMON or an argument) - this would be an automatic array and would require stack allocation. The size of the routine itself should not be relevant. But if you can't figure it out, send us the sources to this routine (and anything needed to compile it) and we'll take a look.

Steve

Retired 12/31/2016

Steve,
I downloaded and installed V6.5A, recompiled everything and it did not help. However, I found the two local arrays (which you said were the automatic arrays) in the subroutine CONSTR2, and put them in the module and made them allocatable arrays, and it appears to have worked. I'm currently running a test with very huge arrays, and it did not give me the stack overflow error yet, so I'm quite sure that worked. THANK YOU very much.

Steve,
Just wanted to let you know it definitely worked to remove the local arrays and put them in a module and make them allocatable. I'm looking forward to when the compiler can handle heap allocation of local arrays.

Glad to hear it.

Steve

Retired 12/31/2016

... it definitely worked to remove the local arrays and put them in a module and make them allocatable. I'm looking forward to when the compiler can handle heap allocation of local arrays.

Note that local variables in a routine can also have the allocatable attribute. It is not required to put them in a module. So, in that sense the compiler can handle heap allocation of local arrays.

-John

Gurus:
I am having this exact same problem. I have a matrix called "D" which is D(NCFD,5) where NCFD can be 1 million+. When I call a sort function with this D matrix, I get the stack overflow. Reading your messages I understand that I need to somehow prevent a temp variable from being created but have no clue as to how to do this. This is the function where the error occurs:

SUBROUTINE SortMatrix(D,NCFD)
IMPLICIT NONE
!Input Variables
INTEGER, INTENT(IN) :: NCFD
!Output Variables
REAL, DIMENSION(0:NCFD-1,0:5), INTENT (INOUT) :: D

The overflow ooccurs as soon as you enter the sub. For this case, NCFD=1001168. Any ideas as to how to change this function so that as temp variable is not created?? Thanks in advance!

Antonio L. Negron

John, You are absolutely correct. I thought moving the local array out into a module or a COMMON block was what solved the problem (not making the array allocatable). However, as you said, you CAN have local arrays, you just need to make them allocatable, which prevents the compiler from making a temp copy of the array on the stack.

Antonio,
You'll have to test a couple of different ways to handle it, but you've either got to put your array "D" in a COMMON block or module, or make it a local allocatable array, with the size being NCFD. i.e.
REAL, ALLOCATABLE:: D(:,:)
.
.
ALLOCATE (D(NCFD,5),STATE=IERR)

Antonio,
In that allocate statement, there's a typo, it should be STAT, not STATE. STAT just returns the status of the attempted ALLOCATE statement, so you can do error handling.

Just a note, Antonio's array is a dummy argument so it cannot be allocatable, unless one doesn't mind waiting for the next version of the Fortran standard (2004?).

In Antonio's case, one option is to make the dummy ary D, an assumed shape array

REAL, DIMENSION(:, :), INTENT (INOUT) :: D ! requires explicit interface (use or host association, or interface block)

Familiarity with the online docs, particularly the following section may be helpful:

Programmer's Guide | Performance: Making Programs Run Faster | Use Arrays Efficiently | Passing Array Arguments Efficiently

hth,
John

John,

If Antonio's array was a dummy argument, then we wouldn't be having this discussion.... Two local arrays were the culprits, and these can be replaced with ALLOCATABLEs.

Steve

Retired 12/31/2016

Guys:

Thanks to all for the help. I still can not get this to work. Reading the docs, I understand that the temp copies are being created because I am sending an assumed shape array to a function with an explicit-shaped array.

My problem is that I do not know how to tell the compiler to receive the assumed shape array "D" into the local subroutine! I have been reading the docs, but I am quite confused about the whole thing.

I am using interface blocks but the compiler complains about the following statement:

    SUBROUTINE CalculateDistances (IDC,XC,YC,ZC,TC,NCFD,D) 
	IMPLICIT NONE 
                     	!!! Inputs to sub 
	INTEGER, INTENT(IN) :: NCFD 
	REAL, DIMENSION (0:NCFD-1), INTENT(IN) :: IDC,XC,YC,ZC,TC 
	 
	!!! Outputs from sub 
	!REAL, DIMENSION (0:NCFD-1,0:5), INTENT(OUT) :: D 
	REAL, ALLOCATABLE :: D(:,:) 
 

The compiler does not like this last statement. At this point I am clueles as to what to do! :o(

I've always used subprograms with arrays the same way I am doing for the
IDC,XC,YC,ZC,TC variables in this function. I think this is forcing the compiler to make temp copies of these variables. I never had problems until now that I am dealing with huge matrices.How can I do this otherwise??

AARGH (Must be monday!)

Again, thanks for your help. I will read the docu (again) to see if it starts making any sense.....

Hi Steve,

There are two sub-threads to this thread. :-)

jschomaker had the local arrays that were causing problems. Making them allocatable will resolve that.

Antonio's array (D) is a dummy arg

 
SUBROUTINE SortMatrix(D,NCFD)  
  IMPLICIT NONE  
  !Input Variables  
  INTEGER, INTENT(IN) :: NCFD  
  !Output Variables  
  REAL, DIMENSION(0:NCFD-1,0:5), INTENT (INOUT) :: D 

Antonio, all you have to do is make the array arguments to CalculateDistances be assumed shape arrays. See my earlier post for an example. I gotta run right now, so maybe someone else can post a version of your routine that uses assumed shape array args.

hth,
John

Take off the ALLOCATABLE attribute from the declaration of D. Currently, dummy arguments can't be ALLOCATABLE. They CAN be deferred-shape, however, which is what you want.

Most important, an explicit interface for CalculateDistances must be visible to its caller, so that the array is passed the right way.

Steve

Retired 12/31/2016

Another possible solution to Antonio's problem may be simply to use his original code and upgrade the compiler. Compaq seems to have made significant progress in the last year in addressing the problem of making unnecessary copies of arrays during subroutine calls. If the assumed shape actual argument will always be contiguous in his application, I think current versions of CVF won't do the copy-in/copy-out that older versions did. The easiest way to find out would be to send the original code to someone with the most recent version of the compiler (Compaq's Fortran support team seems a likely candidate) and have them compile it and see if the copies still get made.

Leave a Comment

Please sign in to add a comment. Not a member? Join today