Fee.dll terribly slows down the debugger

Fee.dll terribly slows down the debugger

Portrait de ovz

Hi All,

My team works on a large code base a significant part of which is in Fortran. We recently upgraded to Intel(R) Visual Fortran Composer XE 2013 Update 3 Integration for Microsoft Visual Studio* 2012, 13.0.3624.11. After this we all are experiencing unbearable slowdowns when debugging the code. Whenever we try to debug Fortran or even just have a Fortran module on the call stack every debugger step through the code takes 10 or more seconds of full CPU load. Using Sysinternals Process Explorer I was able to see the call statck like one listed below. Note FEE.dll which is Intel(R) Fortran Expression Evaluator 13.0.2225.2. 

I need help with this urgently as the entire team cannot work normally because of this. 

msdia110.dll!DllGetClassObject+0x2c060
msdia110.dll!DllGetClassObject+0x2c1ff
msdia110.dll!DllGetClassObject+0x2c278
msdia110.dll!DllGetClassObject+0x2b3b7
msdia110.dll!DllGetClassObject+0x3536e
msdia110.dll!DllGetClassObject+0x1093a
FEE.dll!DllGetClassObject+0x205f7
FEE.dll!DllGetClassObject+0x20972
FEE.dll!DllGetClassObject+0xb18e
FEE.dll!DllGetClassObject+0xb4fe
FEE.dll!DllGetClassObject+0x9cc4
FEE.dll!DllGetClassObject+0x505f
FEE.dll!DllGetClassObject+0x655c
FEE.dll!DllGetClassObject+0x760a
FEE.dll!DllGetClassObject+0x7e14
FEE.dll!DllGetClassObject+0x8354
FEE.dll!DllGetClassObject+0x307e2
VSDebugEng.dll!DkmDllEnsureInitialized+0x8169c
VSDebugEng.dll!ProcDkmFree+0x1989
VSDebugEng.dll!ProcDkmFree+0x1776
VSDebugEng.dll!DkmDllEnsureInitialized+0x857e5
VSDebugEng.dll+0xc25d
VSDebugEng.dll!DkmDllUninitialize+0x24a
ntdll.dll!RtlInitializeExceptionChain+0x85
ntdll.dll!RtlInitializeExceptionChain+0x58

73 posts / 0 nouveau(x)
Dernière contribution
Reportez-vous à notre Notice d'optimisation pour plus d'informations sur les choix et l'optimisation des performances dans les produits logiciels Intel.
Portrait de Sergey Kostrov

>>...Whenever we try to debug Fortran or even just have a Fortran module on the call stack every debugger step through
>>the code takes 10 or more seconds of full CPU load...

By the way, it is a well know problem when debugging C/C++ applications which require lots of memory for processing. There are three possible reasons:

- Application allocates lots of memory on the stack
- Application allocates lots of memory from the system heap
- Stack Reserve and Stack Commit values are too big

Portrait de ovz

Sergey,

I appreciate the advise, but this is not the situation I described. The sluggishness in fee.dll happens _always_ whenver Fortran modules are on call stack, no matter how much memory is consumed etc. C/C++ with no Fortran debugs fine.

Portrait de Sergey Kostrov

I see. Here is another question:

>>
>>...msdia110.dll!DllGetClassObject+0x2c060
>>...

Are you using DIA SDK ( Debug Interface Access SDK ) to get some information from program database files ( *.pdb )?

Portrait de ovz

No, we don't. We don't have any funcitonality that talks to debugger.

Portrait de Sergey Kostrov

>>...Whenever we try to debug Fortran or even just have a Fortran module on the call stack every debugger step
>>through the code takes 10 or more seconds of full CPU load...

Is there any additional output in Output Window of Visual Studio when you're debugging? Please carefully verify.

Portrait de ovz

No, just the usual stuff in Output Window. When I stepping through the code with 10s delay I can definitely see that nothing gets printed.

Portrait de Sergey Kostrov

I would also try commenting some blocks of codes. Sorry, it is really unclear what else could be wrong.

Portrait de Annalee (Intel)

Could you try setting "Tools > Options > Text Editor > Fortran > Advanced > Disable Database" to true?

Portrait de ovz

Thanks for the suggestion Annalee. Disable Database has no effect on the phenomenon though. 

Portrait de iliyapolak

Hi ovz,

did you try to debug with windbg?

Portrait de iliyapolak

There is also possibility to debug a debugger.It can be done by attach invasively to VS process.I see that you were able with process explorer to identify and collect debugger's thread so it could not be a problem to do it with windbg.Next option after locating the VS debugger's thread will be execute !runaway metacommand in order to isolate most active threads in term of cpu time consumption.Next try to put breakpoint of functions that you suppose are responsible for hogging cpu.You can also be forced to disassemble those function to better understand what has happend.

Portrait de iliyapolak

I would also advise before attaching windbg at least try to profile with Xperf VS process while debugging your application.Pay attention to FEE.dll

Portrait de IanH

Debugging the Fortran integration to Visual Studio in the absence of any debug symbols or source is a completely absurd proposition for an Intel Visual Fortran end user.

In the absence of being able to create a simple reproducer I'd be zipping your entire solution up and uploading it as a Premier Support issue.

Portrait de ovz

I did try debugging simple Fortran projects. There no slowdown stepping through it. Our codebase is indeed large and we have large number of lines in files that are in the call stack. It used to work fine in VS 2010 though. Hopefully there's a way to configure Fee.dll not to invoke things it does invoke on every step through the code in debugger. 

Portrait de Annalee (Intel)

Submitting your program to us as a test case would be very helpful. Since it is very large, I would suggest using Premier Support. If you post the issue number, I can assign it to myself.

Annalee

Portrait de Sergey Kostrov

>>>>...Whenever we try to debug Fortran or even just have a Fortran module on the call stack every debugger step
>>>>through the code takes 10 or more seconds of full CPU load...
>>
>>Is there any additional output in Output Window of Visual Studio when you're debugging?

Could you post a complete content of Output Window for the working cycle of your application? Thanks in advance.

Portrait de iliyapolak

>>>Absence of any debug symbols>>>

Maybe FEE.dl has public debug symbols.Process explorer was able to resolve exact function name DllGetClassObject  called from FEE.dll I suppose that the problem is located further in call chain.At least you can inspect return values of callee.DllGetClassObject returns void pointer to interface on failure it will return null pointer.I think that this could be the case of extensive calls to DllGetClassObject.

Portrait de ovz

Annalee, thanks for being ready to help. I need to get an approval to post the codebase to Premier support issue.

Portrait de ovz

Sergey, 

I'm attaching the output in two files. the _BEFORE_ file has output captured before application hits the breakpoint in Fortran. There was a 10 second slowdown after the breakpoint hit but before the enviroment is responsive. Then I stepped through 3 lines, each step assoicated with a delay. _AFTER_ file contains the rest of the output. Again, the output looks totally normal for our applicaiton.

Fichiers joints: 

Portrait de Sergey Kostrov

Thanks for these Output Window files and I'll take a look tomorrow.

Portrait de Sergey Kostrov

>>...There was a 10 second slowdown after the breakpoint hit but before the enviroment is responsive...

Could you try to press Break All from Debug menu during these 10 seconds? And then take a look at a Call Stack.

>>...We recently upgraded to Intel(R) Visual Fortran Composer XE 2013 Update 3 Integration for Microsoft Visual Studio* 2012,
>>13.0.3624.11...

Does it mean that with Initial, Update 1 and Update 2 Releases everything was working without 10 secs delays?

Portrait de ovz

Sergey,

I cannot break the execution during those 10 seconds. Visual Studio is totally unresponsive during this. The slowdown happend after our applications gets stopped for debugging.

We upgraded from Visual Studio 2010 and 

Intel(R) Visual Fortran Package ID: w_fcompxe_2011.9.300
Intel(R) Visual Fortran Composer XE 2011 Update 9 Integration for Microsoft Visual Studio* 2010, 12.1.3526.2010, Copyright (C) 2002-2012 Intel Corporation

To Visual Studio 2012 with full Intel Parallel Studio XE 

Microsoft Visual Studio Professional 2012
Version 11.0.60315.01 Update 2

Intel(R) Visual Fortran Package ID: w_fcompxe_2013.3.171

Intel(R) Visual Fortran Composer XE 2013 Update 3 Integration for Microsoft Visual Studio* 2012, 13.0.3624.11, Copyright (C) 2002-2013 Intel Corporation
* Other names and brands may be claimed as the property of others.

Intel® Advisor XE 2013 Update 2
Intel® Advisor XE 2013 Update 2, (build 270011), Copyright © 2009-2013 Intel Corporation. All rights reserved.

Intel® C++ Composer XE 2013 Package ID: w_ccompxe_2013.3.171
Intel® C++ Composer XE 2013 Update 3 Integration for Microsoft* Visual Studio* 2012, Version 13.0.1207.11, Copyright © 2002-2013 Intel Corporation. All rights reserved.
* Other names and brands may be claimed as the property of others

Intel® Inspector XE 2013 Update 5
Intel® Inspector XE 2013 Update 5, (build 278112), Copyright © 2009-2013 Intel Corporation. All rights reserved.

Intel® VTune™ Amplifier XE 2013 Update 5
Intel® VTune™ Amplifier XE 2013 Update 5, (build 274450), Copyright © 2009-2013 Intel Corporation. All rights reserved.

Portrait de iliyapolak

You need to debug your program under windbg.I stated it earlier.First chance exception is excpected to be handled.Exception is thrown at code executing inside KernelBase32.dll at address  0x000007FBD76B811C and it seems that exception originated at code location or it is location of exception context 0xA1A01DB2.

My advise is to run your program under windbg and use automated analysis with the !analysis -v metacommand.You will not proceed further with VS debugger.Before setting up windbg ensure that you have connection to debug symbol server.

Portrait de iliyapolak

While issueing !analyze-v command you will get a exception context record created by RaiseException function(can be obtained with .ecxr command also).Next step will be trying to understand the exact type of exception.Context contains usualy 3 parameters of exception one of them will be an exception address.Yo can use ln "one of exception parameters" command.If it is resolved successfully to exception address you can dump exception itself with dt command.

In your case it is first chance exception so I would advise to break with debugger on first chance.You can enable this with sxe -eh command.

Portrait de ovz

Thanks for suggestions, Illya. I'm aware of those exceptions and they are not the concern I'm seeking help with. The slowdown of Visual Studio happens when applicaiton itself is already stopped in debugger.

Portrait de iliyapolak

If your concern is perceived slowdown of VS debugger I gave you a two suggestion already.One is simply profile VS process with Xperf you will get nice breakdown of VS thread(s) activity with detailed call stack information and time spent in each functions.

Portrait de IanH

Zitat:

iliyapolak schrieb:

If your concern is perceived slowdown of VS debugger I gave you a two suggestion already.One is simply profile VS process with Xperf you will get nice breakdown of VS thread(s) activity with detailed call stack information and time spent in each functions.

And how is that going to help the OP?  Profiling/debugging/whatevering the Fortran integration in Visual Studio is a job for the Intel devs!

Portrait de iliyapolak

So the best option is to wait for Intel devs to resolve the problem:)

Portrait de Sergey Kostrov

>>...You need to debug your program under windbg.I stated it earlier...

No, this is totally not needed. That was told already many times that Visual Studio has integrated Debugger and it allows debugging at source codes level. Unfortunately, we see that you don't understand this and continue suggesting to everybody '...Use WinDbg..Use WinDbg...Use WinDbg....'. Are you OK? Are you doing any programming with Visual Studio and Fortran?

Portrait de Sergey Kostrov

Ovz,

You have more than 100 First-chance exceptions in the application, like:
...
First-chance exception at 0x000007FBD76B811C in SPM700_d_protected.exe: Microsoft C++ exception: GroveHandle::exception at memory location 0x0000000000137898.
First-chance exception at 0x000007FBD76B811C in SPM700_d_protected.exe: Microsoft C++ exception: GroveHandle::exception at memory location 0x0000000000137898.
First-chance exception at 0x000007FBD76B811C in SPM700_d_protected.exe: Microsoft C++ exception: GroveHandle::exception at memory location 0x0000000000137898.
First-chance exception at 0x000007FBD76B811C in SPM700_d_protected.exe: Microsoft C++ exception: GroveHandle::exception at memory location 0x0000000000137898.
First-chance exception at 0x000007FBD76B811C in SPM700_d_protected.exe: Microsoft C++ exception: GroveHandle::exception at memory location 0x0000000000137898.
...
So, something really goes wrong and you need to try to understand what causes it. Another tip is, you could try to disable All exceptions in the Visual Studio when debugging and take a look at 'Debug' menu.

Portrait de Sergey Kostrov

And one more comment:
...
'SPM700_d_protected.exe' (Win32): Loaded 'C:\Windows\System32\SHCore.dll'. Cannot find or open the PDB file.
First-chance exception at 0x000007FBD76B811C (KernelBase.dll) in SPM700_d_protected.exe: 0xA1A01DB1 (parameters: 0x00000000007658A6, 0x0000000000763420, 0x0000000000000000).
'SPM700_d_protected.exe' (Win32): Loaded 'C:\Windows\System32\uxtheme.dll'. Cannot find or open the PDB file.
...
However, that exception happens only two times, one is before and another one is after the breakpoint was hit.

Portrait de iliyapolak

.>>>Are you doing any programming with Visual Studio and Fortran>>>

All my C programming is done with Visual Studio, Java programming is done with Eclipse and assembly programming is done with MASM32

In one of the previous threads one of the users on Fortran forum has problem with stack overflow caused by an extensive recursive call I advised the user to use windbg to perform automated analysis and that forum user was able to locate and correct the error.  You are failling to understand that windbg is a lot more powerful than VS debugger and can also debug at source level and in case of user Ovz the problem is related to C++  exception and to slowdown of VS debugger during the program debugging and I do not know how the second problem can be solved when user cannot even continue debugging after breakpoint is hit because environment is not responsive. Moreover Ovz has stated that exceptions are not problem for him.

Portrait de ovz

Illya, I agree that WinDbg is quite powerful. Not all the devs on the team are comfortable with it and we need to keep them productive. I could also do some sleuthing in collaboration with Intel developers, and in this I expect guidance from people who are actually in position to fix the issue.  

Portrait de iliyapolak

Ovz completely agree with you.If you are interested I can provide a helping hand in usage of windbg if not to solve but maybe to understand what went wrong with VS debugger.

Portrait de ovz

Sergey,

I'm well aware of the GroveHandle exception. It is thrown _and_ handled by my code. This solution is there for some time now and never caused a debugger issue. 

I've seend 0xA1A01DB1 for a long time too. I assume this is something being thrown and handled by Windows. 

I get the debugger slowdown right in the beginning of the first Fortran subroutine that I call before any of the exceptions listed in the output happen. I also tried a simple example doing the same thing, but that one debugs fine. 

I'm working on clearing the actual source code to be sent to Premier Support. In the meanwhile any suggestions how to resolve this are very welcome. 

Portrait de ovz

Annalee, 

The issue number I would like you to take a look at is 696480. Could you please take a look and see if we can arrange source code transfer as soon as we can?

Portrait de iliyapolak

Ovz,

you provided call stack of one of the VS thread hence my question are you sure that this thread  belongs to debugger?It seems so.As I stated in one of my previous posts the problem could be related to DllGetClassObject and to functions further in call chain.Perceived slowness of the debugger's thread can some kind of synchronous wait maybe hang while waiting on some processing beign done on behalf of caller.

Just my 0.02 cents:)

Portrait de jimdempseyatthecove

As illyapolak mentions, VS Debugger will not complete a step until it has stopped all application threads. Check the Threads debug window to see if any threads are at highest priority. Note, if you are running OpenMP then Name with __kmp_launch_monitor will be Highest priority and this is expected. If any of your other threads are at Highest priority then this may be part of the issue.

Additionally, when array bounds checking is enabled, then sometimes what you have displayed in your Watch, Locals, and Autos may induce an excessivly large number of bounds checking. Experiment with turning off array index out of bounds testing.

Jim Dempsey

www.quickthreadprogramming.com
Portrait de iliyapolak

Actually windbg should be able to locate quickly the stalled VS debugger thread mainly with !runaway command and setting thread context  and single stepping could  reveal what went wrong with debugger's thread.But this is a task for Intel devs.

Portrait de Sergey Kostrov

>>I'm well aware of the GroveHandle exception. It is thrown _and_ handled by my code. This solution is there for some time
>>now and never caused a debugger issue.
>>
>>I've seend 0xA1A01DB1 for a long time too. I assume this is something being thrown and handled by Windows.

Could you do a simple test? If Yes, just comment that piece of codes that throws exception and verify what happens after that.

If it doesn't resolve that problem there are two possibilities:

- This is some problem introduced by Intel Visual Fortran Update 3
- This is some problem with Debugger of Visual Studio 2012 ( you will need to contact to Microsoft / another long story, unfortunately )

If it does resolve that problem there is just one possibility:

- This is a problem with application's codes ( You've already mentioned that a problem with the exception is old )

So, simply ask yourself "What if something is really wrong with our codes?"

Portrait de iliyapolak

>>>- This is some problem introduced by Intel Visual Fortran Update 3
- This is some problem with Debugger of Visual Studio 2012 ( you will need to contact to Microsoft / another long story, unfortunately )>>>

The problem is mostly related to what you wrote in qouted sentences.

Portrait de Sergey Kostrov

>>...The problem is mostly related to what you wrote in qouted sentences...

There is a difference between your '...mostly related...' and my '...there is just one possibility...' because this is just one still Not proven possibility. So, let's wait for a response from Ovz or Intel software engineers who will look into the problem.

Portrait de iliyapolak

Yes agree with you that we should wait for response.

Portrait de Annalee (Intel)

One of my colleagues who specializes in debugging is working on this issue in premier support.

Annalee

 

Portrait de ovz

I've uploaded the source code project and build instructions to Premier tech support. Waiting for response.

Portrait de AK

We have the same issue in our mixed C++/Fortran application. Installed Composer Update 4, but the problem persists.

Using xperf, I found that FEE.dll uses a lot of CPU (~95%) calling into MSDIA110.dll!CDiaEnumTraversal, which again spends a lot of time in MSDIA110.dll!NameMatcher::match.

Any suggestions on fixes or workarounds for this is very much appreciated.

Portrait de jimdempseyatthecove

>>Using xperf, I found that FEE.dll uses a lot of CPU (~95%) calling into MSDIA110.dll!CDiaEnumTraversal, which again spends a lot of time in MSDIA110.dll!NameMatcher::match.

Do you have a lot of variables in your Watch window(s)?
If so, what happens if you remove all of them? (then set the ones you really need)

Do you have any conditional breaks set?
if so, what happens if you remove (all of) them?

One more stab in the dark, try turning off intelisense (thing that shows contets of variable when hovering mouse over a variable).

Jim Dempsey

www.quickthreadprogramming.com
Portrait de Steve Lionel (Intel)

fee.dll is the Fortran debug expression evaluator. The indications here are that it is calling back into Visual Studio to look up names. We don't typically see a lot of overhead from this.

It looks to me as if you are getting appropriate assistance through Premier Support issue 696480.

Steve
Portrait de AK

Thank you for the hints, but no luck..

Zitat:

jimdempseyatthecove schrieb:

Do you have a lot of variables in your Watch window(s)?

If so, what happens if you remove all of them? (then set the ones you really need)

Nothing in the Watch window (or Memory), still slow.

Zitat:

jimdempseyatthecove schrieb:

Do you have any conditional breaks set?

None

Zitat:

jimdempseyatthecove schrieb:

One more stab in the dark, try turning off intelisense (thing that shows contets of variable when hovering mouse over a variable).

No effect..

Portrait de Sergey Kostrov

This is almost one month old problem and my new question is:

Is it an UI problem or something else?

So, in order to verify if it is some UI issue try to compare what Windows messages are received in your "Slow-To-Debug" project vs. a simple "Fast-To-Debug" project ( let me call it so ) with a Microsoft's utility Spy++. The utility should be located at:

[ VSInstallDir ]\Common7\Tools

Name of the executable is: Spyxx.exe

Pages

Connectez-vous pour laisser un commentaire.