x64 debug build failure with "fit_lookup: Line_seq_number 0000000000000000 was not found"

x64 debug build failure with "fit_lookup: Line_seq_number 0000000000000000 was not found"

I have been struggling with this strange error when building debug version of the our solver using Fortran Compiler XE 13.1.1.171. This issue has already been reported to Intel support,but I have not heard anything from them, and I hope someone here can give me a hint - the issue literately block any of Intel ifort v13.x deployment, if not addressed, we will be stuck to ifort v12.1.

1) Build environment: Fortran Compiler XE 13.1.1.171 hosted on VS2010 (part of the Intel Parallel Studio) on Windows

2) Target: x64 only (with or without -openmp), debug build only

3) Build error:

Compiling with Intel(R) Visual Fortran Compiler XE 13.1.1.171 [Intel(R) 64]... Creating temporary file "RSP1.rsp" with contents [ /nologo /debug:full /Od /fpp /I".." /I"../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\lib\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\chem\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /assume:nosource_include /DDEBUG /reentrancy:threaded /extend_source:132 /real_size:64 /align:rec16byte /align:qcommons /align:sequence /fpe:0 /fpconstant /names:lowercase /iface:cref /assume:underscore /module:"../modules/x64/Double Debug/" /object:"x64\Double Debug/" /traceback /check:bounds /check:uninit /libs:static /threads /dbglibs /c /extfor:f /Qvc10 /Qlocation,link,"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\\bin\amd64" "C:\build\V740\source\cnt\Src\cnt_summ.f" ] Creating command line "ifort @"C:\build\V740\source\cnt\x64\Double Debug\RSP1.rsp"" fit_lookup: Line_seq_number 0000000000000000 was not found fortcom: Fatal: There has been an internal compiler error (C0000005). compilation aborted for C:\build\V740\source\cnt\Src\cnt_summ.f (code 1) Creating temporary file "RSP1.rsp" with contents [ /nologo /debug:full /Od /fpp /I".." /I"../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\lib\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\chem\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /assume:nosource_include /DDEBUG /reentrancy:threaded /extend_source:132 /real_size:64 /align:rec16byte /align:qcommons /align:sequence /fpe:0 /fpconstant /names:lowercase /iface:cref /assume:underscore /module:"../modules/x64/Double Debug/" /object:"x64\Double Debug/" /traceback /check:bounds /check:uninit /libs:static /threads /dbglibs /c /extfor:f /Qvc10 /Qlocation,link,"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\\bin\amd64" "C:\build\V740\source\cnt\Src\cnt_dlay.f" ] Creating command line "ifort @"C:\build\V740\source\cnt\x64\Double Debug\RSP1.rsp"" CNTLIB - 1 error(s), 0 warning(s)

4) Reproducibility:

4.1) For above file, it always fails with the same error output, but only limited to x64 debug build. Win32 build is fine, x64 release buid is fine.

4.2) It could appear with a few of other files as well. But the file could be different depending on my build option (for example, /openmp build may have a different set of files with such errors)

4.3) Now the most mysterious part: if I change the build option a little bit, for example, removing one of /traceback /check:bounds /check:uninit for this file, and manual compile will be fine, but if I restored the /traceback /check:bounds /check:uninit setting as before, manual build would NOT fail anymore. Clean up the whole project and build the whole solution again, the failure will appear again.

The last observation seems to indicate the error is related to some build procedures or environment.

Have anyone seen such a behavior before? Our experience with ifort v13.x has been bad for other crucial issue at the runtime. But teh latest bug build failure is just another thing that becomes a show stopper.

BTW, I have the same issue on Linux build as well where it seems /check:bounds /check:uninit /check:pointer all 3 options will be the trigger to the error. The x64/Windows is a bit different, so far I have not yet found a pattern, except the strange behavior in 4.3.

Any suggestion would be welcome.

 

32 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

It looks like the copy of text from build log is scrambled. So here is the error output:

3>Compiling with Intel(R) Visual Fortran Compiler XE 13.1.1.171 [Intel(R) 64]...
3>cnt_summ.f
3>fit_lookup: Line_seq_number 0000000000000000 was not found
4>------ Build started: Project: MECLIB2, Configuration: Double Debug x64 ------
3>fortcom: Fatal: There has been an internal compiler error (C0000005).
4>Compiling with Intel(R) Visual Fortran Compiler XE 13.1.1.171 [Intel(R) 64]...
3>compilation aborted for C:\build\V740\source\cnt\Src\cnt_summ.f (code 1)

Please provide us with a ZIP of the source in question.  You can attach it here, use "Send author a message" to provide it to me privately, or use Intel Premier Support. Please be sure to include any include files or module sources it needs.

You may want to try turning off "Check Routine Interfaces" under Diagnostics to see if that makes a difference. The reproducibility issues you note make me suspect it is relevant.

Steve - Intel Developer Support

Hi, Steve,

There is already a test case (MSVS project) submit to Intel Premier Support (IssueID : 697019). It was originally created for similar Linux build failure, but the reproducibility issue there prevented me to create a test case for isolated file.

I just check the "Check Routine Interfaces", it is off all the time.

Let me know what else I can help.

In addition to the "fit_lookup: Line_seq_number 0000000000000000 was not found" error output, there is another type of error also showing up from time to time:

4>This application has requested the Runtime to terminate it in an unusual way.
4>Please contact the application's support team for more information.
4>GEM_LO_GET_LOCATOR_INFO: zero locator value
4>C:\Users\ALLEN~1.GTI\AppData\Local\Temp\72763.i: catastrophic error: **Internal compiler error: abort signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report.  Note: File and line given may not be explicit cause of this error.
4>compilation aborted for C:\build\V740\source\eng\src\eng_fmac.f (code 1)

"GEM_LO_GET_LOCATOR_INFO: zero locator value" feels more un-predictable if not the same.

But the behavior observed in 4.3) is applicable here. If I remove /traceback, the compiler crash is gone, restore the setting, the compiler crash is not coming back. Totally bizarre.

Remove your option of choice such that you can build.

Perform Clean

Perform Build All

Do you receive an error report indicating one of the .mod files cannot be found?
(Do not re-issue Build until error goes away and be satisfied that the Build is OK).

Circular dependencies in .mod files sometimes gives wierd results. Fix the circular dependency. Then restore options that gave you problems. Then Clean, and Build All and see if problem comes back.

Jim Dempsey

www.quickthreadprogramming.com

Jim,

I doubt we have any .mod circular dependencies: we have several other configurations (for example: Debug | Win32), they just build fine, never reporting any errors with your suggested steps.

Steve,

What really bugs me at this stage is the observation that this Debug|x64 issue only happens on my PC. With the same source codes, I can not even reproduce it on a different machine that also have ifort 12.1 and 13.1 installed. Of course, my PC has a lot of demo and extra compiler installed (I have 12.1, 13.0, 13.1 and 14.0 all installed).

I even did a diskcheck on my pc to see if that helps. Well, it didn't. Some registry messed up or stuff that we do not know? Not sure.

In cases such as this I usually suggest rebooting into "Safe Mode" and see if you can reproduce the problem. If not, then that suggests some background process is interfering with the compilation. You can then use msconfig and the Startup tab to selectively disable startup options and see if you can identify the culprit.

Steve - Intel Developer Support

What Steve is suggestion (though not explicitly stated) is the situation where the share mode of a compiler output/input/temp file is such that when an Anti-Virus opens the file for scanning, that this interferes with the compilation process. This had been a reported problem (which may not have been fixed in the version of the compiler/linker you are using). This may not only be Anti-Virus. Once you identify the culprit, Intel software support may be able to engineer a fix.

Jim Dempsey

www.quickthreadprogramming.com

If it is an external program interfering, there is likely nothing we can do about it. My experience is that problems that can be reproduced on one computer only are due to some influence outside the compiler product. I have even seen a case where a coding error in a printer driver caused problems!

Steve - Intel Developer Support

Just tested the idea with safe-mode (without networking), the problem persists, both errors "fit_lookup: Line_seq_number 0000000000000000 was not found" and "GEM_LO_GET_LOCATOR_INFO: zero locator value" are encountered, and what is interesting and puzzling is that file set that having above errors are different from those observation when building in normal mode.

Just found out ifort 2013.4.190 was released several days ago.

So I went through an excise that I uninstalled all Intel components (Parallel Studio, all compiler update) from my PC, and then re-installed only following:

1) ifort 2011.4.196 (this is by far the only version that works!)

2) All latest Parallel Studio components (ifort/icc/icpc 2013.4.190, VTune 2013 update 7, Inspector Update6, and Advisor Update 3).

Well, the bad news is that on this system only 2 version of ifort exist, the same old errors still persist.

I am really not sure what is causing this now. I think tomorrow, I will have my PC reformated to factory setting, and start from scratch.

Not that there should be anything wrong with this, your .rsp file contains:

C:\build\V740\source\modules\../modules/x64/Double Debug/

While it should not matter if your path goes up and down ("modules\../modules"), do your working configurations have that peculiarity?

The above syntax may require the modules folders to contain the symbolic link (..). In some cases this is not always true.

Jim Dempsey

www.quickthreadprogramming.com

Jim,

I do not recall we ever use .. symbolic link. The build project has been working for more than 8 years for Win32, and 1 year for x64 build; it is just my observation, the ifort is taking either / or \ for path separator, never giving me trouble.

For the output I have shown, the project is having "..;..\modules\$(PlatformName)\$(ConfigurationName)\" or "..;../modules/$(PlatformName)/$(ConfigurationName)/" under Fortran->General->Additional Include Directories and Fortran->Output Files->Module Path.

I just changed the / to \ for this particular project, but somehow the command line display still has C:\build\V740\source\modules\../modules/x64/Double Debug/, not sure how this happened.

The bottom line, after I changed / to \ under Fortran->General->Additional Include Directories and Fortran->Output Files->Module Path, nothing changed for the build, same error for cnt_summ.f.

It you suspect an AV interefering with compilation process I think that booting into safe mode will not help.Filter drivers belong to AV file scanning could be marked as a boot drivers.

Most AV programs have an option to temporarily disable scanning (or manually turn off/on). Barring that you may be able to (temporarily) specify your output folders (temp, obj, mod, ...) as not to be scanned. If AV is not the issue then set everything back.

Jim Dempsey

www.quickthreadprogramming.com

I think AV is not an issue: we have the same failure for roughly the same file sets when running in windows safe mode (without anything running)

just uninstall the Symantec endpoint protection entiredly from my PC, I still run into the same issues. So it is definitely not an AV issue.

BTW, did I forget to mention that on Linux (RHEL5.8), we are seeing the similar type of errors with ifort v13.1? We definitely have no AV there at all. And we found out on Linux, when debug build is using "-check pointer -check bounds -check uninit" together we will be hit by the error

fit_lookup: Line_seq_number (nil) was not found

I have always seen these troubles are related.

I don't think it's related to an external program; I have seen this error in previous releases when Something Bad happens during the time after the compiler starts up and before all its internal initializations have happened.  

For this particular source file, I got it to compile by removing the requirement for /fpp (commented out the #if / #endif).   You'll have to create the correct workaround of course; maybe use !dir$ if / !dir$ endif pairs instead?

I'm still working at getting it to reproduce in an environment that I can debug the compiler.

                    --Lorri

taking off /fpp for this cnt_summ.f does make the compile going through.

But the effect of /fpp behaves like what I have described in 4.3) in the first post. Restoring /fpp, the file still build fine. One of those things that make me pull my hair....

Quote:

jimdempseyatthecove wrote:

Most AV programs have an option to temporarily disable scanning (or manually turn off/on). Barring that you may be able to (temporarily) specify your output folders (temp, obj, mod, ...) as not to be scanned. If AV is not the issue then set everything back.

Jim Dempsey

 That is true did not think about this option.

Just an update:

I have my PC restored, and then installed VS2010, and then ifort 12.1, and then all the latest components of the parallel studio xe 2013.I am doing this to see whether the trouble is from the many installation (some of the Intel beta over the last 2 years) of the intel parallel studio.

The verdict: the reported problem persists, but with a different set of files!

My PC is using Intel i7-2600 CPU @ 3.4GHz with 16GB.

A similar configured PC (in terms of VS-Parallel Studio) which DO not any the issue is using Intel i7-3770 CPU @ 3.4GHz with 8GB.

Is this a software issue or hardward caused issue?  I do not fully understand the cause of the trouble anymore. But this surely will hit us as the whole company. Typically, if the compiler testing does not pass my test, then no one in our company will initiate the update. So this is very frustrating, because if we could not iron this out, our next software release will be frozen to the ifort 12.1 for the whole release cycle (we have a policy that we do not update the compiler once a major release goes to 'gold', and the compiler version will be frozen throughout the whole life cycle of that release.)

I wonder if you have a memory problem. That's really the only thing I can think of at this point. It might be interesting to try removing half the memory and see if the symptom changes or goes away. If it's still there, swap with the other half.

Steve - Intel Developer Support

Could be also some hardware problem which can have a catastrophic influence on compiler output.

Steve,

Just did the memory swap in/out test, the crash persists! So it is not memory.

I just asked another developer (who also have the same Sandybridge CPU and Windows 7) to test the project build with latest ifort, he got the same compiler crash:

2>------ Build started: Project: VTRLIB, Configuration: Double Debug x64 ------
2>Compiling with Intel(R) Visual Fortran Compiler XE 13.1.2.190 [Intel(R) 64]...
2>vtr_phas.f
3>------ Build started: Project: MECLIB2, Configuration: Double Debug x64 ------
3>Compiling with Intel(R) Visual Fortran Compiler XE 13.1.2.190 [Intel(R) 64]...
3>mec_sol2.F
2>fit_lookup: Line_seq_number 0000000000000000 was not found
2>fortcom: Fatal: There has been an internal compiler error (C0000005).
2>compilation aborted for C:\s\mihail_solver_v7.4.x\source\vtr\src\vtr_phas.f (code 1)
2>vtr_popv.F
4>------ Skipped Build: Project: lapack, Configuration: Double Debug x64 ------
4>Project not selected to build for this solution configuration
5>------ Skipped Build: Project: xblas, Configuration: Double Debug x64 ------
5>Project not selected to build for this solution configuration
3>fit_lookup: Line_seq_number 0000000000000000 was not found
3>fortcom: Fatal: There has been an internal compiler error (C0000005).
3>compilation aborted for C:\s\mihail_solver_v7.4.x\source\mec\src\mec_sol2.F (code 1)
3>mec_jbrgm.f

I will have to reach out to other developers to test the latest ifort to see how widespread this issue is and if there is any pattern, but from all it seems, this would be a serious road block now.

>>>3>fortcom: Fatal: There has been an internal compiler error (C0000005).>>>

It looks like access violation error and can be caused indirectly by faulty hardware.

Nope,

I have widen the deployment testing throughout the company's f90 developpers, so far all those who have tried the latest Intel ifort have all reported back the same errors! I am talking about more than 10 F90 developers (using mostly Dell workstation with Sandy Bridge CPU and a few Ivy Bridge CPU)!

An interesting observation from yesterday testing is this: the file sets which had the compiler crash are the same! I am pretty sure they are all having the same source code snapshot. So the previous reproducibility problem (see the 4) in first post) actually seems very deterministic with the same source codes.

This is definitely not a hardware issue, I am going to report to Intel support and request a deeper look. Because this is really hitting us big time or we will have to stay away the whole ifort v13.x release cycle.

Is this still Premier Support issue 697019?

Steve - Intel Developer Support

Steve,

It is still the Premier Support issue 697019 right now.

I think I will have to open a new one considering the importance of issue. Issue 697019 was reported for a similar Linux compiler crash for debug build only, but ultimately I found a work around (i.e., remove one of the options: -check pointer -check bounds -check uninit). The work around is not desirable, but still acceptable. But then we found Windows compiler has the similar crash, all further report was then lumped in.

I think I would report it separately as an Windows ifort issue, instead of letting all those stuck in a Linux issue report.

Ok, that's a good idea. Please let me know the new issue number.

Steve - Intel Developer Support

We believe that this problem got fixed for compiler version 14.0 in September. Let us know if you're still seeing it.

Steve - Intel Developer Support

Regarding an error description in some cases a misaligned stack (no. of arguments pushed is not equal to no. of arguments poped) could be responsible for access violation exception.

Login to leave a comment.