Building fails in a large Visual Studio 2013 solution using parallel project builds and multiprocessor compilation because tempo

Building fails in a large Visual Studio 2013 solution using parallel project builds and multiprocessor compilation because tempo

Is there a known issue with building Fortran projects in parallel in Visual Studio 2013?

When I choose to “Rebuild Solution” the build will fail in ifort for seemingly random Fortran projects with  “error #10104: unable to open 'C:\Users\{user name}\AppData\Local\Temp\{temporary file name}’”

The temporary file names appear to have the format {4 digit number}tempfile{index number}.

After this build failure, if I choose to “Build Solution” most of the Fortran projects that failed to build in the “Rebuild Solution” operation will build okay, but I may get another error #10104. After the second build failure if I again choose to “Build Solution” the remaining Fortran projects build okay and I get a complete build of my software.

While as a developer I can handle the “Rebuild Solution” failure without much issue by initiating 2 or 3 “Build Solution” passes, our automated build process cannot (and should not need to). I would like to minimize build times using parallel project builds and multiprocessor compilation for our automated builds but it does not seem to work very well.

I am using Visual Studio 2013 and Intel(R) Visual Fortran Composer XE 2013 SP1 Update 1 Integration for Microsoft Visual Studio* 2013, 14.0.0074.12.
My computer’s guts are two Intel Core i5 CPU M 560 @ 2.67Ghz. I have four processors. 
My solution contains 60 projects. 28 of them are Fortran projects.  The rest are c++ projects.
In Visual Studio\Tools\Options\Projects and Solutions\Build and Run I have a 4 specified for “maximum number of parallel project builds.”
For all 28 of my Fortran projects I have the Property Pages\Fortran\General\Multi-processor Compilation set to “Yes (/MP)”

If I set the maximum number of parallel project builds to one, I do not get the error #10104.

If I set the maximum number of parallel projects to 4 and set the multi-processor compilation to “No” for all of my Fortran projects, I do not get the error #10104.

The other developers on our team also have this issue on their development environments.

I suspect parallel project building and multi-processor compilation is causing ifort to try to open/use temporary files with the same name. Is there a way to me to work around this problem and keep the benefits of parallel project building and multi-processor compilation?

15 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

I'm not aware of this being a current problem, though I've seen somewhat similar reports in the past. Here's an experiment - temporarily disable your antivirus software and see if the error remains. Can you attach a ZIP of one of the buildlog.htm files that shows the error?

Here's an alternative you can try - instead of using parallel project builds. enable the Fortran > General > Multiprocessor Compilation option and see if that gives you improved build times/

Steve - Intel Developer Support

Are you using the preprocessor?   If so, try adding /Qsave-temps to the command line.   I don't believe there is a property for this, so you'd have to use the "Command Line" tab.

                    --Lorri

When I turned off my anti-virus’ active-protection I no longer get the error #10104 and the solution builds with zero errors.  Our anti-virus software is VIPRE 6.0.0 from GFI Software, Ltd.

Steve, to clarify, I get this error when I use both parallel project builds and Fortran > General > Multiprocessor Compilation option set to “Yes (/MP)”. Disabling either one of these features will allow the build to complete. However, having both of these features enabled significantly improve build times so I’m endeavoring to keep them both switched on.

I have attached a BuildLog.htm for a project that fails with error #10104.

With the anti-virus active protection turned on, I tried Lorri’s suggestion to add /Qsave-temps to the command line. The solution’s build still fails but slightly differently. Instead of ‘error #10104: unable to open {tempfilename}’, the output window shows:
Can't open file {tempfilename} for write
ifort: error #10298: problem during post processing of parallel object compilation
compilation aborted for {source code filename} (code 1)

I feel stuck in the void between two magic black boxes. What/where is ifort doing stuff that the anti-virus should ignore when the anti-virus is doing whatever it is doing? I guess I would like to know the information necessary about how ifort works to configure the anti-virus software to leave the compiling/building process alone.

 

Attachments: 

AttachmentSize
Download BuildLog1_0.zip4.21 KB

Most antivirus programs have a means to specify exceptions. You can usually specify a folder, as well as a file with wild cards. I use Avast Antivirus.

Open the Avast User Interface
Settings
Active Protection
File System Shield (tool button)
Exclusions

In there I have added "C:\test\*\RCX*.tmp"

I chose not to add the .exe's

Your AV may have a similar facility.

Jim Dempsey

www.quickthreadprogramming.com

Yep - I tell my AV to not do auto-scanning of my project folder. I have often seen AV products hold open newly created files for scanning, thus causing problems for build systems.

Steve - Intel Developer Support

Thank you all for the feedbacks! After excluding the project file directories from the anti-virus active scan the solution is able to build successfully without errors related to write-protected temporary files. Now off to navigate the politics of enabling developers to muck with AV settings.

Unfortunately, we still have this problem intermittently (every 5th or 6th attempt) on our build system with the build failing with the error 
error #10104: unable to open 'C:\Users\{user name}\AppData\Local\Temp\{temporary file name}

Will an Intel guru please confirm this issue and create an official issue? I would really like to use the /MP switch on the build system (Significant improvement in compile times) but using the /MP switch seems too unreliable currently.

The build system does not have antivirus software that actively scans files.

While turning off the antivirus scanning helps mitigate the issue in my development environment, the issue still pops up every once in a while. That's okay for my development environment, but it isn't for the build environment.

I have attached an example solution that demonstrates the problem with every build I try, even with the antivirus disabled. It has 100 projects that each compile 100 (small) source files. The projects are built in parallel (Visual Stuido 2013 Tools - Options - Projects and Solutions - Build and Run - maximum number of parallel project builds = 4) and the source files are compiled in parallel (the /MP switch in Project Properties - Configuration Properties - Fortran - General - Mutli-processor Compilation).

If you try to build the attached solution (MultiProcessorParrallelBuildCheck.sln) using more than one processor, you will probably get the issue (especially if your antivirus software's active scanning is turned on). On my development environment, about 30% of the projects do not build with anti virus scanning turned on. 

With the antivirus disabled, more projects compile okay, but about 5% of the projects will still fail to build.

Attachments: 

AttachmentSize
Download MultiProcBuildIssue.zip164.23 KB

I was able to make a failure happen with this, but it took several tries. Let me see what I can find out. Issue ID is DPD200255678.

Steve - Intel Developer Support

Thank you very much for confirming this issue Steve. I'm looking forward to continued improvement in the /MP functionality.

I think it is a reasonable recommendation for users to disable antivirus software activities to maximize compilation/building speed. However, my preference would be for the compilation/build process to work okay even with these antivirus services enabled (with the expectation of build sluggishness). In my company several users of the Fortran compiler will not be able to modify antivirus settings without IT intervention. It winds up being a significant waste of time for us.

The major issues with AV programs is that they sometimes view the creation of EXEs as a behavior of malware, and will stop the process or delete the EXE once you build it. There's nothing we can do about that. We can't also control background processes that block access to files.

In this particular case, I think it is that the part of the compiler that manages multiprocessor compilation isn't sufficiently protecting against reuse of the filename (which is composed in part of the process ID). That is something we can work on.

Steve - Intel Developer Support
Best Reply

We have fixed the problem reported here and we thank Nathan for providing a test case we could work with. I expect the fix to appear in Update 4 to Composer XE 2013 SP1.

Steve - Intel Developer Support

Good for the fix. If there is a lingering AV vs .EXE issue, and due to IT department policy you cannot muck with the AV settings, you might these two hacks

1) Specify the output executable name as YourProgramName.duh. Then add as an after build pass a "COPY *.duh *.exe". Note the use of COPY in place of REN(ame) as either may be caught by the AV and get deleted.

2) Specify the output executable name as YourProgramName.BAT. That might slip by the AV. This has the quirky advantage that you can run the file by issuing "YourProgramName" or as named with .bat. Yes, it is not a batch file, Windows doesn't care about the name for running, it inspects the first few bytes of the file to determine what to do. If it looks like an executable it will load like an executable. All the .BAT or .EXE do is modify the file search path decisions.

Edit: Additional information

If you have an issue with IT policy vs AV the IT department should have no qualms about installing Virtual PC as it is sandboxed in a virtual environment thus providing the require AV protection from the Host perspective. Within the Virtual PC, the contained O/S can run without AV. Should your Intel software license preclude you from installing into the Virtual PC (you would not necessarily want to install twice), you then have the option of performing your builds in the host environment to the *.duh files, then from the Virtual PC "Copy H:\*.duh *.exe" where H: is a network mapped drive on the Virtual PC to the development folder on the host.

Jim Dempsey

www.quickthreadprogramming.com

The problem in this case was not AV-related. The names of temporary files used to collect output from the parallel compiles were not "unique enough".

Steve - Intel Developer Support

I’m grateful for the fix; the /MP option in conjunction with parallel project builds makes a big difference in the build times for large solutions with many projects. I’m looking forward to Update 4 to Composer XE 2013 SP1.

I have not (yet) run into the issue with AV deleting output executable files but I appreciate Jim’s workarounds to what Steve identified as a major issues with AV software. I’m sure the information will help somebody! 

Leave a Comment

Please sign in to add a comment. Not a member? Join today