Large array IO optimization

Large array IO optimization

I have large array (8x3000000 is typical) from data acq system to process. Unfortunately, the req'd processing can be accomplished in one pass so the entire array must be held in memory..I can allocate an array for the whole thing and do an unformatted read, but that ties up the machine for the entire i/o question is What is the most efficient way to do the read piecewise to allow for an occasional progress indicator update and perhaps a user request to cancel the operation?

So far, I've discovered that if I ALLOCATE buf(nChn, nScans), the READ(ioun) buf takes several times as long as the write did under Visual Basic. This, of course, is risky business, but the data acq box is an old 90MHz P-I while the processor is a 200+MHz PII....I had expected the FORTRAN read to be at least on the same order as the VB PUT....however, this may just be demonstrating the weakness of IDE vs SCSI as the data acq box has a brand new UW SCSI adapter/drive whereas the analysis box is run of the mill brand D notebook. Both are running NT4, if that has any bearing on it.

Anybody have suggestions of proper coding form to at least minimize the pain? I'm going to go try a direct fixed allocation although all data was not collected under the same conditions so variable sizes will have to be handled eventually.

2 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

If I recall from your posting on comp.lang.fortran, you are using FORM='BINARY' for your Fortran I/O. Try adding BUFFERED='YES' to the OPEN - that should help a lot.


Retired 12/31/2016

Leave a Comment

Please sign in to add a comment. Not a member? Join today