Linking takes too long

Linking takes too long

I've just converted from Absoft to Intel as my FORTRAN compiler. The only thing I don't like is that it takes several minutes to re-compile. Specifically, the compiling stage goes very quickly, but after the Output says "Linking...", it takes several minutes. I'm using the MKL library for FFTs and two small libraries that I wrote, one in FORTRAN, the other in C. Does anyone know why the linking takes so long and if there's a way to speed up the link process? The Absoft compiler took just a few seconds for the same task. Thanks.

37 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

If you are comparing link speed of one compiler without link time optimization with a compiler which (optionally) performs such optimization as a prelude to linking (Microsoft /GL or Intel IPO) you may expect differences. For example, in the Visual Studio properties for Intel compilers, there are 3 levels of interprocedural optimization: none, within a source file, or link time IPO.

Steve Lionel (Intel)'s picture

It's not normal for linking to take a long time, unless you have a very large application or are using the IPO feature, as Tim mentions. Does this happen for any application, including a simple "Hello World"?

Steve
jimdempseyatthecove's picture

Other than for IPO as tim18 mentions, there was a thread mentioning an issue where the developer was on a client in a client/server situation and where tools and libraries were located (and/or paths pointed to) the server. I cannot recall if the problem had to do with client authentication or some other Managed Network issue. Maybe this will jog someones memory and they can point out the forum post.

Jim Dempsey

www.quickthreadprogramming.com

Thanks to all for the suggestions.

I found the IPO option, and it is set to "No".

I'll set up the simplest of programs and see if that takes a long time to link.

I am running the compiler in a networked situation where perhaps some other program is objecting to what the linker has to do. I read about someone who had firewall issues that caused slowness in some stage of the process.

Steve Lionel (Intel)'s picture

What exactly is "a networked situation"? Is the compiler and Visual Studio installed on your PC's local hard drive? Are you using projects on a network drive? Is your home directory on a network drive?

Steve

Quoting - Steve Lionel (Intel)
What exactly is "a networked situation"? Is the compiler and Visual Studio installed on your PC's local hard drive? Are you using projects on a network drive? Is your home directory on a network drive?

No, nothing like that. All the code and libraries are local on a PC running Windows XP. A colleague who has helped me with getting ifort running has the same issue.

We use the HDF4 libraries. I wonder if there's something about them that would take a long time to link to.

Quoting - evankw21
I've just converted from Absoft to Intel as my FORTRAN compiler. The only thing I don't like is that it takes several minutes to re-compile. Specifically, the compiling stage goes very quickly, but after the Output says "Linking...", it takes several minutes. I'm using the MKL library for FFTs and two small libraries that I wrote, one in FORTRAN, the other in C. Does anyone know why the linking takes so long and if there's a way to speed up the link process? The Absoft compiler took just a few seconds for the same task. Thanks.

I made a "Hello World" project, and it linked in no time. I then added my HDF call and all the things required to do it, and it still links quickly. So it looks like it's not the HDF library that's doing it to me. The projects that take a long time to link are not trivial. One of them that I looked at has 42 subroutines. Would this be enough to make the linking take several minutes?

My next suspect would be the MKL library. I link to a library of mine that uses MKL_DFTI.

If anyone has any more thoughts, I'd appreciate it. Waiting several minutes to link is really making it difficult to debug programs.

Steve Lionel (Intel)'s picture

No, 42 subroutines is trivial. Turn on the Linker property page "Show progress messages: Show some progress messages". This will show you how it is finding various libraries and you may be able to identify the sluggard. My guess is that the linker is searching network shares.

Steve

Quoting - Steve Lionel (Intel)
No, 42 subroutines is trivial. Turn on the Linker property page "Show progress messages: Show some progress messages". This will show you how it is finding various libraries and you may be able to identify the sluggard. My guess is that the linker is searching network shares.

Steve:

I tried what you asked. After saying, "Linking...', it took about 90 s before the rest of it came out, and when the rest of it came out, it only took a second or two. I'll try the very verbose option next.

Evan

1>------ Build started: Project: abxp, Configuration: Debug Win32 ------
1>Compiling with Intel Visual Fortran 11.1.035 [IA-32]...
1>abx_init.f90
1>Linking...
1>Searching libraries
1> Searching E:EvanifortlibHDF42r3_sourcehdf4libreleasedllhd423m.lib:
1> Searching E:EvanifortlibHDF42r3_sourcehdf4libreleasedllhm423m.lib

.....

1> Searching C:Program FilesIntelCompiler11.135mklia32libmkl_core.lib:
1> Searching C:Program FilesIntelCompiler11.135libia32libiomp5mt.lib:
1> Searching C:Program FilesMicrosoft SDKsWindowsv6.0Alibuuid.lib:
1>Finished searching libraries
1>LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:library
1>Embedding manifest...
1>Performing Post-Build Event...
1>1 file(s) copied.
1>
1>Build log written to "file://E:EvanifortabxpDebugBuildLog.htm"
1>abxp - 0 error(s), 1 warning(s)
========== Build: 1 succeeded, 0 failed, 3 up-to-date, 0 skipped ==========

Quoting - evankw21

Steve:

I tried what you asked. After saying, "Linking...', it took about 90 s before the rest of it came out, and when the rest of it came out, it only took a second or two. I'll try the very verbose option next.

Evan

1>------ Build started: Project: abxp, Configuration: Debug Win32 ------
1>Compiling with Intel Visual Fortran 11.1.035 [IA-32]...
1>abx_init.f90
1>Linking...
1>Searching libraries
1> Searching E:EvanifortlibHDF42r3_sourcehdf4libreleasedllhd423m.lib:
1> Searching E:EvanifortlibHDF42r3_sourcehdf4libreleasedllhm423m.lib

.....

1> Searching C:Program FilesIntelCompiler11.135mklia32libmkl_core.lib:
1> Searching C:Program FilesIntelCompiler11.135libia32libiomp5mt.lib:
1> Searching C:Program FilesMicrosoft SDKsWindowsv6.0Alibuuid.lib:
1>Finished searching libraries
1>LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:library
1>Embedding manifest...
1>Performing Post-Build Event...
1>1 file(s) copied.
1>
1>Build log written to "file://E:EvanifortabxpDebugBuildLog.htm"
1>abxp - 0 error(s), 1 warning(s)
========== Build: 1 succeeded, 0 failed, 3 up-to-date, 0 skipped ==========

So now I set the verbose option to full. This time I timed in on my watch, and it was a little over 120 s to complete. Still, almost all the time was spent after the "Linking ..." command. I looked at Task Manager, and it said that fortcom.exe was using a single CPU. After a couple minutes I noticed that a task called debenv.exe took over from fortcom.exe. Here was the start of the printout:

1>------ Build started: Project: abxp, Configuration: Debug Win32 ------
1>Compiling with Intel Visual Fortran 11.1.035 [IA-32]...
1>abx_init.f90
1>Linking...
1>Starting pass 1
1>Processed /DEFAULTLIB:ifconsol
1>Processed /DEFAULTLIB:libifcoremt
1>Processed /DEFAULTLIB:libifport
1>Processed /DEFAULTLIB:libmmt

Does that shed any light?

Evan

Quoting - evankw21

So now I set the verbose option to full. This time I timed in on my watch, and it was a little over 120 s to complete. Still, almost all the time was spent after the "Linking ..." command. I looked at Task Manager, and it said that fortcom.exe was using a single CPU. After a couple minutes I noticed that a task called debenv.exe took over from fortcom.exe. Here was the start of the printout:

1>------ Build started: Project: abxp, Configuration: Debug Win32 ------
1>Compiling with Intel Visual Fortran 11.1.035 [IA-32]...
1>abx_init.f90
1>Linking...
1>Starting pass 1
1>Processed /DEFAULTLIB:ifconsol
1>Processed /DEFAULTLIB:libifcoremt
1>Processed /DEFAULTLIB:libifport
1>Processed /DEFAULTLIB:libmmt

Does that shed any light?

Evan

Steve:
Just to make sure, I compiled the Release configuration, and the same thing happened. "Linking..." got printed very quickly, then fortcom.exe popped up in the Task Manager and used a CPU for 80 s, then disappeared. devenv.exe, which had been present but using no CPU, then started using the entire CPU for another 40 s or so. Then the print statements came pouring out for about 15 s. Then it said it was creating the Manifest file, then it finished about 5 s later.
Evan

Steve Lionel (Intel)'s picture

Mysterious. Some additional things to try.

First, I'll ask that you download and install 11.1.048 and try the build again. If it still takes a long time, do this experiment.

Open the vcbin subfolder of your Visual Studio installatioin. Rename link.exe to linkx.exe. Find notepad.exe on your system (probably in Windows), copy it into vcbin and rename it to link.exe. Now try linking again. This will fail, but I'm interested in knowing how long it takes before you see notepad come up with an error message. Once this is done, delete the fake link.exe and rename the old one back.

Steve

Quoting - Steve Lionel (Intel)

Mysterious. Some additional things to try.

First, I'll ask that you download and install 11.1.048 and try the build again. If it still takes a long time, do this experiment.

Open the vcbin subfolder of your Visual Studio installatioin. Rename link.exe to linkx.exe. Find notepad.exe on your system (probably in Windows), copy it into vcbin and rename it to link.exe. Now try linking again. This will fail, but I'm interested in knowing how long it takes before you see notepad come up with an error message. Once this is done, delete the fake link.exe and rename the old one back.

Steve:

Could you give me a link to the 11.1.048 download. I've been clicking around for 10 minutes, but can't find it.

Thanks.

Evan

Steve Lionel (Intel)'s picture

Log in to the Intel Registration Center - and the link should be there for you.

Steve

Quoting - evankw21

Steve:

Could you give me a link to the 11.1.048 download. I've been clicking around for 10 minutes, but can't find it.

Thanks.

Evan

My sysadmin was able to get the newest version and I then had to re-enter stuff under Tools => Options => IVF => Libraries and Includes.

It still took a long time after "Linking ..."

I then renamed the link.exe and replaced it with a link to notepad.exe. It still said "Linking ..." and waited a long time before coming back with the expected error:

Error 1 error #10036: unable to run 'link' Link

So why could it be taking so long just get to the linker???

Evan

David White's picture

Quoting - evankw21

My sysadmin was able to get the newest version and I then had to re-enter stuff under Tools => Options => IVF => Libraries and Includes.

It still took a long time after "Linking ..."

I then renamed the link.exe and replaced it with a link to notepad.exe. It still said "Linking ..." and waited a long time before coming back with the expected error:

Error 1 error #10036: unable to run 'link' Link

So why could it be taking so long just get to the linker???

Evan

Evan,

Just a long shot here -- I had compile time issues when I am connected to our network via VPN because Visual Studio by default puts files in My Documents, which in our scenario are on a network drive. Do you have any settings pointing to a network drive. Similarly, when searching for the linker, if it is searching through PATH and there are any network drives before the location of the linker, then you could be waiting for these slow connections. Try putting the linker and compiler folders towards the start of PATH and see if that makes any difference.

Regards,

David

jimdempseyatthecove's picture

Evan,

David's VPN post was the one I referenced in my prior message.

Jim

www.quickthreadprogramming.com
Steve Lionel (Intel)'s picture

Can you try a command-line build and see if the problem is still there? I think you've identified that the delay is before the linker is actually invoked, but I can't think of what it might be, especially when some of your projects link quickly.

Oh, I thought of another experiment. Open the folder C:Program FilesIntelCompiler11.146binia32 and rename xilink.exe to xilink2.exe. Open the VCBIN folder as above and copy link.exe into the compiler's binia32 folder and rename the copy to xilink.exe. Test your build again.

Steve

Quoting - Steve Lionel (Intel)
Can you try a command-line build and see if the problem is still there? I think you've identified that the delay is before the linker is actually invoked, but I can't think of what it might be, especially when some of your projects link quickly.

Oh, I thought of another experiment. Open the folder C:Program FilesIntelCompiler11.146binia32 and rename xilink.exe to xilink2.exe. Open the VCBIN folder as above and copy link.exe into the compiler's binia32 folder and rename the copy to xilink.exe. Test your build again.

Steve:

That makes the Linking very fast!! Do you know what the problem was, then, and why? Is there a more standard fix than renaming the files like that?

Evan

Quoting - David White

Evan,

Just a long shot here -- I had compile time issues when I am connected to our network via VPN because Visual Studio by default puts files in My Documents, which in our scenario are on a network drive. Do you have any settings pointing to a network drive. Similarly, when searching for the linker, if it is searching through PATH and there are any network drives before the location of the linker, then you could be waiting for these slow connections. Try putting the linker and compiler folders towards the start of PATH and see if that makes any difference.

Regards,

David

David/Jim:

I don't think this is the issue because everything I have is local. Thanks anyway.

Evan

Steve Lionel (Intel)'s picture

Do I know what the problem is? No. Do I know where it is? I have a better idea.

Some explanation - xilink is the Intel "prelinker" that is primarily used when you have asked for "whole program interprocedural optimization" (/Qipo). It is looking for Intel intermediate language in objects, combining them and then running the optimizer. After doing that, it calls the linker.

Even though you are not using IPO, xilink still needs to read all of the objects and libraries to see if there is IP code in there, and I suspect it is this that is causing the slowness. Was there some particular object or library which, when you added it to the list, caused the slowdown? I don't remember offhand.

Steve

Quoting - Steve Lionel (Intel)
Do I know what the problem is? No. Do I know where it is? I have a better idea.

Some explanation - xilink is the Intel "prelinker" that is primarily used when you have asked for "whole program interprocedural optimization" (/Qipo). It is looking for Intel intermediate language in objects, combining them and then running the optimizer. After doing that, it calls the linker.

Even though you are not using IPO, xilink still needs to read all of the objects and libraries to see if there is IP code in there, and I suspect it is this that is causing the slowness. Was there some particular object or library which, when you added it to the list, caused the slowdown? I don't remember offhand.

I haven't started paring the code down to see which part is the culprit. I did Hello World and added a single HDF routine call, and that was still OK.

BTW, I'll be out of town for a week, so I won't be able to do any more detective work after today.

Steve Lionel (Intel)'s picture

Ok. When you do find the culprit, if you can attach a ZIP of it here that would be helpful.

Steve

Quoting - Steve Lionel (Intel)
Ok. When you do find the culprit, if you can attach a ZIP of it here that would be helpful.

Steve:

The culprit seems to be in mkl_dfti, specifically DftiCreateDescriptor. Here is my code, with some comment lines saying what happens to me when I do certain things.

Evan

program Console1
implicit none
integer :: nfft=16
real :: pser(16)
complex :: spec1(9)

call r2c_fwd_fft(pser,4,spec1)
! If above call is commented out, the program links quickly. If present, linking takes several minutes

print *, 'Hello World'

end program Console1

subroutine r2c_fwd_fft(rts,nord,cspec)

use MKL_DFTI
! If above line is commented out, I get:
! Error 1 error #6457: This derived type name has not been declared. &
! [DFTI_DESCRIPTOR] E:EvanifortConsole1Console1.f90 48
! I copied mkl_dfti.f90 from "D:Program FilesIntelCompiler11.148mklinclude" to the source directory for this
! project

! If I don't have the Fortran => Libraries => Use Intel Math Kernel Library set to "Sequential...", then I get:
! Error 1 error LNK2019: unresolved external symbol _dfti_create_descriptor_1d &
! referenced in function _R2C_FWD_FFT. Console1.obj

implicit none

real :: rts(:)
integer :: nord,nfft
complex :: cspec(:)
type(DFTI_DESCRIPTOR), POINTER :: My_Desc1_Handle
Integer :: Status(6)=0

nfft=2**nord
Status(1) = DftiCreateDescriptor(My_Desc1_Handle, DFTI_SINGLE, DFTI_REAL, 1, nfft)
! This is the line that causes the linker to be very slow. Without the above line, but with
! the following 5 lines, the linker is fast.

!Status(2) = DftiSetValue( My_Desc1_Handle, DFTI_PLACEMENT, DFTI_NOT_INPLACE)
!Status(3) = DftiSetValue(My_Desc1_Handle,DFTI_CONJUGATE_EVEN_STORAGE,DFTI_COMPLEX_COMPLEX) ! Default, optional
!Status(4) = DftiCommitDescriptor(My_Desc1_Handle)
!Status(5) = DftiComputeForward(My_Desc1_Handle, rts(1:nfft), cspec(1:nfft/2+1))
!Status(6) = DftiFreeDescriptor(My_Desc1_Handle)

if (sum(Status) /= 0) then
print*,'r2c_fwd_fft Warning: sum(Status(1:6)) /= 0 from MKL'
print*,' Status(1:6) = ',Status(1:6)
endif

return
end subroutine

Steve Lionel (Intel)'s picture

Thanks - I can see the behavior you describe and will try to figure out what is going wrong.

A comment - your test program is incorrect because an explicit interface to r2c_fwd_fft needs to be visible to the caller. However, it does allow the problem to be seen - I imagine you cut this down from a larger program.

Steve

Quoting - Steve Lionel (Intel)
Thanks - I can see the behavior you describe and will try to figure out what is going wrong.

A comment - your test program is incorrect because an explicit interface to r2c_fwd_fft needs to be visible to the caller. However, it does allow the problem to be seen - I imagine you cut this down from a larger program.

Steve:

Hope you find a solution. It seems like this would have come up for others.

About your second comment, I read up on explicit interface. Would making the r2c_fwd_fft subroutine part of a module make the code correct? When you said "incorrect", did you mean it wouldn't work or that it was not good coding practice? I'm not as familiar with creating interfaces as with using modules, so I could use a reference to a tutorial on that. Thanks.

Also, I read about the flag /check:arg_temp_created that you referenced in one of your articles on argument passing. Is this available as a Fortran option? I couldn't find it under Properties for the project. If it's not there, how do you put it into the compile statement? When I started converting to the use of modules, I started to get compiler complaints about passing an array starting location, like x(1,1,jsub,jfreq) into a subroutine that was expecting an array. I've "fixed" it by changing the actual argument to x(:,:,jsub,jfreq), which compiles and works. But I'm hoping it is not causing a copy of the data to be made.

Evan

Steve Lionel (Intel)'s picture

By incorrect I mean that the program won't execute properly because the compiler does not know thast it needs to pass a descriptor for the assumed-shape array. Yes, putting the routine in a module would be a fine solution.

/check:arg_temp_created is under the "Runtime" property page - warn when array argument uses temporary storage, or something like that.

It would be ok to pass an array element to an array unless the array you are passing is a POINTER array. If it is ALLOCATABLE, that's ok.

Steve

Quoting - Steve Lionel (Intel)

By incorrect I mean that the program won't execute properly because the compiler does not know thast it needs to pass a descriptor for the assumed-shape array. Yes, putting the routine in a module would be a fine solution.

/check:arg_temp_created is under the "Runtime" property page - warn when array argument uses temporary storage, or something like that.

It would be ok to pass an array element to an array unless the array you are passing is a POINTER array. If it is ALLOCATABLE, that's ok.

Steve:

I hope you haven't given up on this. I'm still suffering through waiting two minutes every time I recompile. Thanks.

Evan

Steve Lionel (Intel)'s picture

I haven't given up but I did need to put it aside. What I did find is that the "slow" version pulls in a lot more code than the "fast" version of your code. (I didn't see any difference as to whether your contained function was called or not.) What's odd is that trying to reproduce this in a smaller test case doesn't show a difference, so I'm not yet sure what specifically is the trigger.

I'll eventually need to involve the MKL folks, so you may want to skip the middleman and report this in the MKL section - they may be able to help you sooner.

Steve

Quoting - Steve Lionel (Intel)
I haven't given up but I did need to put it aside. What I did find is that the "slow" version pulls in a lot more code than the "fast" version of your code. (I didn't see any difference as to whether your contained function was called or not.) What's odd is that trying to reproduce this in a smaller test case doesn't show a difference, so I'm not yet sure what specifically is the trigger.

I'll eventually need to involve the MKL folks, so you may want to skip the middleman and report this in the MKL section - they may be able to help you sooner.

Steve:

I owe you an apology and a beer.

Back when it was suggested I turn off IPO, I said that I checked it and that it was already off. Well, apparently, I found the IPO setting under the Fortran tab and verified that it was off. But after starting the topic in the MKL Forum and getting the same advice, I checked again and found that the Linker IPO setting had indeed been set to Yes. I guess I checked the IPO flag back when I wasn't as familiar with all the tabs and sub-tabs for the Project Properties.

Sorry to have wasted your time on that. I'm curious, though, were you able to recreate my 2-minute linking problem? If so, what was causing yours? I assume you had your IPO set to No.

Evan

Steve Lionel (Intel)'s picture

In my project, IPO is not set. I can reproduce the long link time (on my system it's more like 30 seconds, but with the "fast" code the linking is just a second or two.

Steve

Quoting - Steve Lionel (Intel)
In my project, IPO is not set. I can reproduce the long link time (on my system it's more like 30 seconds, but with the "fast" code the linking is just a second or two.

I've heard from experts, in a similar case, that the IPO setting doesn't influence the pre-link stage search for IPO objects. It may be possible to turn off that search (taking responsibility for absence of IPO objects), but not in the same (undocumented) way for different compiler versions.
I too would find this useful, particularly when performing a binary search for a bug in a single .o file among thousands. Don't try it on a laptop disk.

Quoting - Steve Lionel (Intel)
In my project, IPO is not set. I can reproduce the long link time (on my system it's more like 30 seconds, but with the "fast" code the linking is just a second or two.

Steve:

By "fast code" do you mean the code with the call to the DFTI routine commented out?

My original "slow" code that had the problem now takes just a second or two.

Evan

Steve Lionel (Intel)'s picture

Slow with the call to DftiCreateDescriptor, Fast with the calls to the other five Dfti routines in its place. For reasons not yet clear to me, calling DftiCreateDescriptor pulls in a lot more stuff.

Tim is correct that xilink is always invoked, whether or not IPO is enabled. It is xilink that is taking so long, the MS linker is not as sluggish. Unfortunately, there's no option you can set (that I know of) to skip the use of xilink.

Steve

Hi Steve,

I'm having the same problem. I use intel visual composer xe 2013. I used the same trick that you mentioned but it didn't work. I have to keep the original xilink.exe so that the linking works. Any solutions for this long time linking (up to 4 minutes).

Steve Lionel (Intel)'s picture

If you can provide us with a test case we'll be glad to investigate.

Steve

Login to leave a comment.