Error - input statement requires too much data

Status Points: 100 (Total Points: 2,300)

Wee Beng T.

Tue, 08/21/2012 - 00:44

Hi,

I got this error when I try to read the data :

forrtl: severe (67): input statement requires too much data, unit 2, file /home/Results/7/u_value.txt

I did "od -t x4 u_value.txt |head" and got:

0000000 00134100 3f8036d8 3f8089eb 3f80cc25
0000020 3f80fdd4 3f81258b 3f814646 3f8160c3
0000040 3f817565 3f8184da 3f81900d 3f8197ef
0000060 3f819d56 3f81a0f0 3f81a341 3f81a4aa
0000100 3f81a570 3f81a5c6 3f81a5cf 3f81a5a4
0000120 3f81a558 3f81a4f6 3f81a488 3f81a414
0000140 3f81a39d 3f81a328 3f81a2b7 3f81a24a
0000160 3f81a1e2 3f81a180 3f81a125 3f81a0cf
0000200 3f81a080 3f81a036 3f819ff2 3f819fb3
0000220 3f819f79 3f819f44 3f819f13 3f819ee7

What does that mean? I'm suspecting that the file is too large to read in but it's only 60mb of binary data.

Thanks for the help!

10 posts / 0 new

Last post

For more complete information about compiler optimizations, see our Optimization Notice.

Steve Lionel (Intel)

Tue, 08/21/2012 - 07:31

It means that you did an unformatted READ but the file record was shorter than the list of variables to be read into. The file itself isn't "too big to read in". If I am interpreting this right, you have a record that is 1261824 bytes long, which would be 315456 REAL(4) values. How big is the array you are reading into?

Retired 12/31/2016

Izaak Beekman

Thu, 08/23/2012 - 06:46

Was the data written by a fortran program? As Steve alluded, Fortran unformatted IO is written with record identifiers that bound each record and specify the length of the record in bytes. If you try to read in a variable that is longer than the record length in one read statement you'll get this error. Chances are that if you want to read data written with another programming language, then likely you want to be using stream IO which will read binary data directly without record identifiers, as C like languages do.

If you have the code that wrote the file this is the easiest way to figure out how to read it in.

Steve Lionel (Intel)

Thu, 08/23/2012 - 07:30

Given the od output, I'm fairly certain that the file was written by Fortran as a sequential unformatted file. Your advice is still good in general terms.

Retired 12/31/2016

Wee Beng T.

Fri, 08/24/2012 - 04:36

It's only 60mb of binary files. I found that the problem occurs when I running on more than 4 procs. On 4, sometimes it work, sometimes it doesn't. However, I switch to using direct access and now there's no problem. Thanks anyway.

Wee Beng T.

Sun, 08/26/2012 - 03:26

Hi, I found my problem. 1st, I was using "unformatted" instead "binary". 2ndly, I'm now dividing my grid size into different values for each procs. The error occurs when I use MPI_SCATTER to redistribute the values into each procs. Should have used MPI_SCATTERV instead.

However, if I still used "unformatted" instead "binary" after solving the 2nd error, I still get the "Error - input statement requires too much data" why is that so? Is there is difference between "unformatted" and "binary"?

I also tried to solve my problem using direct access. It works for 1 procs. However, increasing procs to 2 results in very big file size (from 94k to 274mb), although it still seems to work (not thoroughly tested). I wonder why.

My code is:

TO SAVE

1st, I save the in sequence, starting from the 1st procs:

size_x*size_y is the total amt of data, jsta/jend is the start/end index for each procs in the y direction

do ij=0,num_procs-1

    call MPI_Barrier(MPI_COMM_WORLD,ierr)

   if (ij==myid) then

       if (myid==0) then

           open (unit = 2 , file = "u_value.txt" , status = "replace", iostat = openstatus(2), form='binary',
            access='direct', recl = size_x*size_y)

        else

           open (unit = 2 , file = "u_value.txt" , status = "old", iostat = openstatus(2), form='binary',
            access='direct', recl = size_x*size_y)

        end if

        write (2, rec = (jsta-1)*size_x+1) ((real(u(i,j),4),i=1,size_x),j=jsta,jend)

       close(2)

    end if

end do

TO READ

do ij=0,num_procs-1

call MPI_Barrier(MPI_COMM_WORLD,ierr)

   if (ij == myid) then

        open (unit = 2 , file = "u_value.txt" , status = "old", iostat = openstatus(2), form='binary',
        access='direct', recl = size_x*size_y)

        read (2, rec = (jsta-1)*size_x+1) ((u(i,j),i=1,size_x),j=jsta,jend)

        close(2)

     end if

end do

So what's wrong with my code? I thought recl should be the total data size while rec is the location to start the data retrival.

Status Points: 15,455 (Total Points: 201,620)

jimdempseyatthecove

Mon, 08/27/2012 - 05:36

>> read (2, rec = (jsta-1)*size_x+1) ((u(i,j),i=1,size_x),j=jsta,jend)

What is jsta?
Is it setup right __for each image__?

e.g. jsta = ij+1

Or did you intend to use:

read (2, rec = ij*size_x+1) ((u(i,j),i=1,size_x),j=jsta,jend)

Jim Dempsey

Wee Beng T.

Mon, 08/27/2012 - 11:50

Let me clarify, i'm reading/writing to a 2d u data.

my total grid size is size_x*size_y

I divide my grid in the y direction, with jsta/jend denoting the start/end index in the y direction.

Hence for a size of 4*8, using 4 procs, i will get :

proc : jsta/end

0: 1/2

1: 3/4

2: 5/6

3: 7/8

My understanding of the "rec" is the starting position to read in the u data.

hence, my "rec" = (jsta-1)*size_x+1), which varies for each procs

Is my usage and reasoning correct? I guess I've made a mistake somewhere.

Steve Lionel (Intel)

Mon, 08/27/2012 - 12:06

Best Reply

RECL= is the number of bytes per record. So, for example, if RECL=10, doing an I/O with REC=5 accesses bytes 40-49 (keep in mind that REC= starts with record 1.) (As a side-note, for FORM='BINARY' the RECL= default is bytes, but for FORM='UNFORMATTED', it's units of 4 bytes by default.)

You may want to consider using the standard syntax FORM='UNFORMATTED', ACCESS='STREAM', and using POS= in your I/O statements. This is better than using the non-standard FORM='BINARY' and better defined.

Retired 12/31/2016

Wee Beng T.

Tue, 08/28/2012 - 09:00

Thanks Steve. I didn't know about "stream" until now. I will give it a try with the new suggestion.

Back to original post

Please sign in to add a comment. Not a member? Join today

Error - input statement requires too much data

Leave a Comment