Compiling with OpenMP directives : debugger provides wrong line numbers

Compiling with OpenMP directives : debugger provides wrong line numbers

Bild des Benutzers SCHWOERTZIG T.

Hi,

I have some difficulties due to OMP directives.  I am compiling on linux64 with ifort 12.1.3.293 build 20120212 and following flags:
 -g -O0 -fp-model precise -openmp  -Dmkl -ftz -extend-source -DFCCI -check uninit -fpe0 -ftrapuv  -traceback

. I am stepping into a subroutine during debug (idbc or gdb). But both debuggers provides exactly the same line number (888) which does not correspond to the called subroutine. The called subroutine is written in a source file which contains 5 subroutines. I am calling the 5th one which start at line 1567. Line 888 is the middle of the 3rd subroutine.

Each subroutine in this source file uses OMP directives :
!$OMP DO SCHEDULE(DYNAMIC,1)
[fortran source]
!$OMP END DO

The problem disappear if I remove these OMP directives. Line number is also consistent with source file.
Is it expected or a known issue ?

Thanks for any help.
Thierry

 

13 Beiträge / 0 neu
Letzter Beitrag
Nähere Informationen zur Compiler-Optimierung finden Sie in unserem Optimierungshinweis.
Bild des Benutzers Lorri Menard (Intel)

It's possible.  We have had some issues in the past with the line number correlation being incorrect when debugging.

I don't see where we've fixed any of those since 12.1, so updating to a new compiler is not likely to help.

I appreciate that you can't share the full project, but can you characterize the program better?  Is line 888 legitimately somewhere in the backtrace?   If not, is that third subroutine (where line 888 lives) also called in the same OMP region as the fifth subroutine?  Any other clues? 

In general, the previous problems were caused by a "stale" line number being reused, and that's what I'd expect was happening here too.

                   Thanks --

                       --Lorri

Bild des Benutzers SCHWOERTZIG T.

Hi Lorri,

Thank you for this feedback.

The situation occurs with 1 threads. Subroutine_5 is called first. but i step directly on line 888.

Furthermore even if i create the breakpoint before running the program with idbc, the line number is not correct (888 again) :

[schwoertzig]$ idbc --args ./my_program -nthread 1
Intel(R) Debugger for applications running on Intel(R) 64, Version 12.1, Build [77.329.14]
------------------
object file name: ...
 Reading symbols from ...done.
(idb) break SUBROUTINE5
Breakpoint 1 at 0x656f38: source.F, line 888.
(idb) bt
The "backtrace" command has failed because there is no running program.

Here are line details from source.F :
Subroutine 1 : line 1-361
Subroutine 2 : line 368-722
Subroutine 3 : line 729-1226
Subroutine 4 : line 1233-1560
Subroutine 5 : line 1567-1688

Here is an extract of the flowchart.

 |   +--SUB_A
 |    |   +--SUB_A
 |    |    |   +--Subroutine 5
 |    |    |   +--Subroutine 4
 |    |   +--SUB_C
 |    |    |   +--Subroutine 3
 |    |   +--Subroutine 1
 |    |   +--Subroutine 2

Do you need more information ?

Thank you and Best Regards
Thierry

 

 

 

Bild des Benutzers SCHWOERTZIG T.

Moving Subroutine5 to a new source file also fixes the issue. Debuggers are able to locate the correct line number corresponding to the subroutine beginning.
Does anybody have an explanation taking into account this new clue ?

Thanks and Best Regards
Thierry

Bild des Benutzers Tim Prince

Are you telling us in-lining affects your debugging?

How about

-debug inline-debug-info

or 

-fno-inline-functions

if you don't want to compile procedures separately?

Bild des Benutzers SCHWOERTZIG T.

Quote:

Tim Prince wrote:
How about
-debug inline-debug-info

Thank you for this idea.
I just tested thses compilation options without success :
  -Ob0
  -debug inline_debug_info
  -inline none
  -inline-debug-info

  • Idbc –args ./myprogram –nthread 1

(idbc) break SUBROUTINE_5

Breakpoint 1 at 0x5f6b08: source.F, line 888

This is still the unexpected line number located in the middle of SUBROUTINE3 (in the same source file).

Thierry

Bild des Benutzers jimdempseyatthecove

Thierry,

Are these subroutines in a module? If so, and I've experienced this myself (and I imagine others here have too), it is possible that the output directory for your .mod and .o files is not the input folder for the link phase of the .o files. In this situation you end up compiling the new code and linking the old code. A quick test for this is to insert a "write(*,*) 'new code'" somewhere in the new code that gets executed early. If this write does not occur, then check your paths as this would indicate you are linking old object files.

Inlining and any optimizations will affect line numbering. A technique I often use is to insert a CALL to a subroutine that is explicitly stated as to not be inlined (and additionally not visible to IPO as it aggressively inlines). The called routine doesn't need to do anything. Usually, the line number where the call is made is always available. Once at break point, you can then single step (and disregard information presented as to line number).

Jim Dempsey

www.quickthreadprogramming.com
Bild des Benutzers SCHWOERTZIG T.

Quote:

jimdempseyatthecove wrote:
Thierry,
Are these subroutines in a module?
...
Jim Dempsey

Hi Jim,
Thank you for this information.
In my case subroutines are not in a module. I also compiled from scratch when i tested the inlining flags.

Using your tick, I added CALL VOID() (on line 1615) at the beginning of the SUBROUTINE5 (which beginns on line 1567).  I made a breakpoint inside VOID SUBROUTINE which is empty. Using "up" command to move in the backtrace debugger says to be on line 935 which is again somewhere in the middle of SUBROUTINE3 which is not related to SUBROUTINE5. This is really enigmatic.

Thierry

 

Bild des Benutzers jimdempseyatthecove

Thierry,

Looks like compiler optimization issue (rearranging code but not line numbers).

One other potential issue is: does the statement following the CALL VOID() contain a "variable" than is a statement function?

Some older programs made use of statement functions. These appear in the data declaration section and when immediately in front of code, are often confused with code.

AVG(A,B,C) = (A+B+C)/3.

The above, when appearing in the data section is a statement function (A,B,C must have been declared).
If the above appears after the first code statement, then it is a statement (not a "statement function").

When debugging, the debugger might hop to the location of the declaration of the statement function (then hop back on next step).

Jim Dempsey

www.quickthreadprogramming.com
Bild des Benutzers SCHWOERTZIG T.

Hi Jim,

Thank you for the idea. There are only Data statments before and no such situation after that. (Replacing data statments does not change the behavior.)

I found a second workaround :
In SUBROUTINE_2 there are 2 !$OMP DO block :

!$OMP DO SCHEDULE(DYNAMIC,1)
      DO IG = 1, NG
      ...
      END DO
!$OMP END DO
C
!$OMP DO SCHEDULE(DYNAMIC,1)
      DO IG = 1, NG
      ...
      END DO
!$OMP END DO

Removing $OMP END DO directives also fixes the issue : Breaking into SUBROUTINE_5 leads to the correct line number (1567 instead of 888).

Thierry

 

Bild des Benutzers jimdempseyatthecove

That should be sufficient information for Intel support to go on, but they are hesitant to do anything without a simple reproducer. If you can put together a simple reproducer then they will likely fix the problem.

Jim Dempsey

www.quickthreadprogramming.com
Bild des Benutzers Kevin Davis (Intel)

I created a much shorter test program based on all the details in this thread but was unable to reproduce the failure so my test is not imitating enough of the original code to produce the line number confusion. IDB is finds/sets the break at the correct line number for subroutine 5 in my case with 14.x and older 12.1 compilers.

Is there any chance at getting a reproducer?

You can provide those in private via a private message or submitting an issue under Intel Premier Support.

Bild des Benutzers Lorri Menard (Intel)

Thank you Jim - couldn't have said it better myself!  :-D

Yes, Thierry, I tried to create my own reproducer given the patterns you've described, with no luck.
I don't need a runnable reproducer, just something I can compile and look at our internal tool that "tracks" line number<->source correlations.

And, you can use the "Private Message" utility to send it along for further security.

           Thank you --

                                     -- Lorri

Melden Sie sich an, um einen Kommentar zu hinterlassen.