Difficult to say without code (I know, your code may be too big or too private to post) but some ideas to consider :
(a) An increase in the creation of temporary arrays in your "few more algebraic statements"
(b) Increased paging - are you accessing matrices in the correct (most efficient) subscript order?
(c) Increased I/O
(d) Antivirus software interference.
(e) Networking traffic.
Les

A couple of other things I noticed.

First - the VTune run was with non-optimized (ie, debug enabled) code. You said there was little-to-no difference between the two program versions with non-optimized code, so finding no difference in VTune is probably expected.

Second - I find it interesting that you're seeing a degradation with your OpenVMS implementation as well. The Intel Fortan compiler and OpenVMS Fortran compiler have the same heritage, but they diverged about 10 years ago and have virtually nothing in common anymore, especially related to optimizations.

Therefore -- I think you should look at some of Les's suggestions in the previous reply. Are you accessing matrices differently now? Are there more temporary arrays? I *believe* that /check:arg_temp_created was supported in the OpenVMS version (it has been 10 years and all) so you may be able to use that to find any new temps.

I hope this helps --

--Lorri

You don't show what TINY is. While the intrinsic names aren't reserved, this usage seems dangerous, particularly if you don't use implicit none. TINY(ZSGWRTAP(2,IL)) might make sense.

Try commenting out just the:

IF (ZSGWRTAP(2,IL).GT.TINY) THEN
ZWSGWRL(IL)=MAX(0.E0,ZWDC2SG(IL)-ZSGWRTAP(1,IL))
FZWSGWRL(IL)=ZWSGWRL(IL)/MAX(TINY,ZSGWRTAP(2,IL)-ZSGWRTAP(1,IL))
ENDIF

statements. Leave the remainder of the code in place.

This particular test in a loop may interfere with vectorization. Should the commented code return the performance then you may need to look at a vector friendly conditional statement.

Also, if IL is your loop control variable, then consider swapping indicies on ZSGWRTAP such that it is

IF (ZSGWRTAP(IL,2).GT.TINY) THEN
ZWSGWRL(IL)=MAX(0.E0,ZWDC2SG(IL)-ZSGWRTAP(IL,1))
FZWSGWRL(IL)=ZWSGWRL(IL)/MAX(TINY,ZSGWRTAP(IL,2)-ZSGWRTAP(IL,1))
ENDIF

Jim Dempsey

In any code where you have convergence, the iterations are highly dependent upon the precision. The algorithm would affect the number of iterations to converge or fail dependent upon the precision... higher precision does not necessarily dictate faster/slower convergence.

It is important to note that the default precision is REAL(4) in IVF. Therefore you should be aware that 1.0 and 123.45E+6 are REAL(4) literals, whereas 1.0D0 and 123.45D+6 are REAL(8) literals (compiler options can change default real size). Therefore

IF(Real8Var .gt. 1.0) then

Is a comparison of a REAL(8) to a REAL(4), which depending upon compile time options may result in runtime code that loads the REAL(4) literal, promotes it to REAL(8), then performs the test. Promotion of literals is not guaranteed at compile time. This is due to maintaining compatibility of results where promotion was only performed at run time.

Jim Dempsey

You must look closely at how you compute the convergence. Not only at the source code level but also in combination with the compiler options relating to expression evaluations (default may have changed). Check/experiment with the /fp:... switches.

You might consider inserting trace printouts during various sections of the convergence:
Iterations 1:100
Iterations 1000:1100
Iterations 10000:10100
Iterations 100000:100100
Iterations: 270000:270100
Iterations 316200:316300
Iterations 327000:327100

(or use spans of +10)

Seeing how the components of what determines convergence vary may yield information as to what is going on and how to fix it.

You may find that your old code was so wrong that correct code may have yielded convergence in a fraction of the number of iterations (and with better results).
Jim Dempsey

Look at the disassembly code for 32-bit Release
do i=1,10000000000 !1e10
00EC108E mov dword ptr [I],1
00EC1095 mov dword ptr [ebp-14h],0
a=a*b
00EC109C movsd xmm0,mmword ptr [A]
00EC10A1 movsd xmm1,mmword ptr [B]
00EC10A6 mulsd xmm0,xmm1
00EC10AA movsd mmword ptr [A],xmm0
enddo
00EC10AF mov eax,dword ptr [I]
00EC10B2 mov edx,dword ptr [ebp-14h]
00EC10B5 mov ecx,1
00EC10BA mov ebx,0
00EC10BF add eax,ecx
00EC10C1 adc edx,ebx
00EC10C3 mov dword ptr [I],eax
00EC10C6 mov dword ptr [ebp-14h],edx
00EC10C9 mov eax,540BE400h
00EC10CE mov edx,2
00EC10D3 mov ecx,dword ptr [I]
00EC10D6 mov ebx,dword ptr [ebp-14h]
00EC10D9 sub ecx,eax
00EC10DB mov dword ptr [ebp-3Ch],ecx
00EC10DE sbb ebx,edx
00EC10E0 mov dword ptr [ebp-38h],ebx
00EC10E3 jl MAIN+9Ch (0EC109Ch)

This being 32-bit code, and using a loop control variable I being 64-bit and using 32-bit regisers, the management of the loop control variables exceed that of the computation portion of the loop.
BTW this loop has an invariant content and therefore this loop should have been eliminated (content executed once).
In looking at the 64-bit code:
do i=1,10000000000 !1e10
000000013F2810A0 mov qword ptr [I],1
a=a*b
000000013F2810A8 movsd xmm0,mmword ptr [A]
000000013F2810AD movsd xmm1,mmword ptr [B]
000000013F2810B2 mulsd xmm0,xmm1
000000013F2810B6 movsd mmword ptr [A],xmm0
enddo
000000013F2810BB mov eax,1
000000013F2810C0 add rax,qword ptr [I]
000000013F2810C4 mov qword ptr [I],rax
000000013F2810C8 mov rax,2540BE400h
000000013F2810D2 mov rdx,qword ptr [I]
000000013F2810D6 cmp rax,rdx
000000013F2810D9 jge MAIN+0A8h (13F2810A8h)
The computation of the expression is the same, but there is less overhead in processing the loop control variable.
This too suffers the optimization issue of not eliminating the loop when the contents are invariant.
...
*** Both loops have an optimization issue where the actual variable I is updated at each iteration (its memory locatio is updated) as oppose to optimizing a temporary I into a register (64-bit) or register pair (32-bit).
..
The above has /O3 - Maximum Speed.
*** However, Intel in their non-Infinite wisdom, chooses not to fully optimize unless you also have enabled Interprocedural optimizations (IPO).
Code generated using IPO
32-bit
a=a*b
00021065 mulpd xmm1,xmm0
00021069 mulpd xmm1,xmm0
0002106D mulpd xmm1,xmm0
do i=1,10000000000 !1e10
00021071 add esi,8
00021074 mov edi,esi
00021076 adc ecx,0
00021079 sub edi,eax
0002107B mov edi,ecx
a=a*b
0002107D mulpd xmm1,xmm0
do i=1,10000000000 !1e10
00021081 sbb edi,edx
00021083 jb MAIN+65h (21065h)
a=1.d0
00021085 movaps xmm0,xmm1
do i=1,10000000000 !1e10
00021088 mov esi,eax
a=1.d0
0002108A unpckhpd xmm0,xmm1
b=1.000000001d0
0002108E mov ecx,2
a=1.d0
00021093 mulsd xmm1,xmm0
00021097 movsd mmword ptr [esp+28h],xmm1
b=1.000000001d0
0002109D movsd xmm0,mmword ptr [___xt_z+74h (0A62A0h)]
000210A5 movsd xmm1,mmword ptr [esp+28h]
do i=1,10000000000 !1e10
000210AB add esi,1
000210AE mov edi,esi
000210B0 adc edx,0
000210B3 sub edi,eax
000210B5 mov edi,edx
a=a*b
000210B7 mulsd xmm1,xmm0
do i=1,10000000000 !1e10
000210BB sbb edi,ecx
000210BD jb MAIN+0ABh (210ABh)
---------------------------------------------------------
64-bit
do i=1,10000000000 !1e10
000000013F24104E xor eax,eax
a=1.d0
000000013F241050 movaps xmm6,xmmword ptr [__xt_z+50h (13F2EB4C0h)]
b=1.000000001d0
000000013F241057 movaps xmm0,xmmword ptr [__xt_z+60h (13F2EB4D0h)]
a=a*b
000000013F24105E mulpd xmm6,xmm0
000000013F241062 mulpd xmm6,xmm0
000000013F241066 mulpd xmm6,xmm0
do i=1,10000000000 !1e10
000000013F24106A add rax,8
a=a*b
000000013F24106E mulpd xmm6,xmm0
do i=1,10000000000 !1e10
000000013F241072 cmp rax,540BE400h
000000013F241078 jb MAIN+5Eh (13F24105Eh)
a=1.d0
000000013F24107A movaps xmm0,xmm6
do i=1,10000000000 !1e10
000000013F24107D mov edx,540BE400h
a=1.d0
000000013F241082 unpckhpd xmm0,xmm6
b=1.000000001d0
000000013F241086 mov rax,2540BE400h
a=1.d0
000000013F241090 mulsd xmm6,xmm0
b=1.000000001d0
000000013F241094 movsd xmm0,mmword ptr [__xt_z+70h (13F2EB4E0h)]
do i=1,10000000000 !1e10
000000013F24109C inc rdx
a=a*b
000000013F24109F mulsd xmm6,xmm0
do i=1,10000000000 !1e10
000000013F2410A3 cmp rdx,rax
000000013F2410A6 jb MAIN+9Ch (13F24109Ch)
enddo
The above vectorized (two doubles (results) per xmm register) and untrolled 4x2 times), then performed remainder, and registerized the loop control.

This dummied up example is not necessarily representable of your application problem (unless your loop control variable is INTEGER(8).
If your application uses INTEGER(8), then I would suggest you code dfferently. Example
INTEGER(4) :: nOuter, nInner,iOuter,iInner
INTEGER(8) :: N, i
...
i=1
nOuter = N / 1000000000_8
do iOuter=1,nOuter
... call yourWork(...)
i=i+1
end do
nInner = MOD(N, 1000000000_8)
do iInnerr=1,nInner
... call yourWork(...)
i=i+1
end do

Sorry I could not embelish the above post with additional coments due to this $%&! forum editor going bonkers when your reply exceeds the size you can stretch this edit box.

Suggestions now are:

Enable IPO in the compiler (at least single file)
*** and enable IPO in the linker.

Jim Dempsey

BTW, in prior to last post (INTEGER(8) loop control variable)
Perform this optimization only if (when) IPO does not reduce the loop control overhead to an acceptible portion of the overall loop iteration.
Also note, in your exampe, i was not used inside the loop, therefore the i=i+1 could be eliminated.
Jim Dempsey

>>Unfortunately, making all integers integer(4) didn't help my run time, or the number of iterations to converge in my release code.
If your integer(4) loop control variables are iterating 2^31 or more times, then this will cause havok in those loops (note, this may not cause a program crash). You will need to assure that should a loop require larger interations, that you consider modifying the code as discussed above (nested loops) or use the integer(8) loop control.

Jim Dempsey