amplxe-gui crashes while Finalizing results / Resolving information for xxx

amplxe-gui crashes while Finalizing results / Resolving information for xxx

Kernel: Linux rbeasley-lx 3.2.0-57-generic #87-Ubuntu SMP Tue Nov 12 21:35:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 12.04.3 LTS

Intel VTune Amplifier XE 2013 Update 15

I just began an evaluation of Intel VTune Amplifier XE 2013 Update 15 for Linux. After sampling my application by attaching to a running process, VTune is unable to ever perform an analysis of the samples. The application always ends up crashing during the "Finalizing results" phase. When crashing, the current library being resolved varies with each attempt.

My machine has 18G of RAM with an add'l 8G of swap.

(gdb) bt
#0 0x00007fb4819c2015 in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#1 0x00007fb4819bc187 in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#2 0x00007fb4819bb6a5 in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#3 0x00007fb4819df54a in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#4 0x00007fb4819df47c in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#5 0x00007fb4819def4a in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#6 0x00007fb4819a874f in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#7 0x00007fb483a16448 in dicerresolver_2_2::internal::ism_module_bank::get_symbol(dicerresolver_2_2::internal::segment_rva const&) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_impl_2.2.so
#8 0x00007fb483a4743d in dicerresolver_2_2::internal::virtual_stack::virtual_stack(dicerresolver_2_2::internal::resolver_impl&, dbinterface_1_38::Index, dicerresolver_2_2::internal::segment_rva const&, boost::shared_ptr<dicerresolver_2_2::internal::module_bank>, dicerresolver_2_2::resolution_type_set&) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_impl_2.2.so
#9 0x00007fb483a325d8 in dicerresolver_2_2::internal::resolver_impl::resolve_code_locations(dicerresolver_2_2::internal::attribute_row_ref_table*, dbinterface_1_38::Index, dbinterface_1_38::Index, boost::shared_ptr<dicerresolver_2_2::internal::module_bank>, msngr2::IProgress*, dicerresolver_2_2::resolution_type_set&, dicerresolver_2_2::internal::call_target_row_ref_table&) ()
from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_impl_2.2.so
#10 0x00007fb483a42c02 in dicerresolver_2_2::internal::resolver_impl::resolve_module(dbinterface_1_38::Index, msngr2::IProgress*, dicerresolver_2_2::internal::call_target_row_ref_table&) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_impl_2.2.so
#11 0x00007fb4837595ad in dicerresolver_2_2::internal::parallel_module_loading::do_resolve(msngr2::IProgress*, dbinterface_1_38::Index, dicerresolver_2_2::internal::call_target_row_ref_table&) const () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_core_2.2.so
#12 0x00007fb483759e41 in dicerresolver_2_2::internal::parallel_module_loading::operator()(tbb::blocked_range<dbinterface_1_38::Index> const&) const () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_core_2.2.so
#13 0x00007fb483775344 in tbb::internal::start_for<tbb::blocked_range<dbinterface_1_38::Index>, dicerresolver_2_2::internal::parallel_module_loading, tbb::auto_partitioner>::execute() () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_core_2.2.so
#14 0x00007fb492ba8911 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libtbb.so.2
#15 0x00007fb492ba66fc in tbb::internal::arena::process(tbb::internal::generic_scheduler&) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libtbb.so.2
#16 0x00007fb492ba583c in tbb::internal::market::process(rml::job&) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libtbb.so.2
#17 0x00007fb492ba335b in tbb::internal::rml::private_worker::run() () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libtbb.so.2
#18 0x00007fb492ba3266 in tbb::internal::rml::private_worker::thread_routine(void*) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libtbb.so.2
#19 0x00007fb4a9d7ee9a in start_thread (arg=0x7fb479222700) at pthread_create.c:308
#20 0x00007fb4a84313fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#21 0x0000000000000000 in ?? ()

(gdb) info share /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_impl_2.2.so
From To Syms Read Shared Object Library
0x00007fb483a0aa60 0x00007fb483a5a048 Yes (*) /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_impl_2.2.so
(*): Shared library is missing debugging information.

(gdb) frame 7
#7 0x00007fb483a16448 in dicerresolver_2_2::internal::ism_module_bank::get_symbol(dicerresolver_2_2::internal::segment_rva const&) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_impl_2.2.so
(gdb) info reg
rax 0x1 1
rbx 0x3b170 242032
rcx 0x1f226 127526
rdx 0x1f21d 127517
rsi 0x7fb55deb53b0 140416941511600
rdi 0x7fb47921ec38 140413103107128
rbp 0x7fb470081ba0 0x7fb470081ba0
rsp 0x7fb47921f040 0x7fb47921f040
r8 0x1 1
r9 0x8 8
r10 0x7fb481c1c368 140413247800168
r11 0x1f21d 127517
r12 0x7fb47921fcb0 140413103111344
r13 0x7fb47921fc80 140413103111296
r14 0x7fb47007bb70 140412950395760
r15 0x7fb47007bb70 140412950395760
rip 0x7fb483a16448 0x7fb483a16448 <dicerresolver_2_2::internal::ism_module_bank::get_symbol(dicerresolver_2_2::internal::segment_rva const&)+264>
eflags 0x202 [ IF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0

publicaciones de 12 / 0 nuevos
Último envío
Para obtener más información sobre las optimizaciones del compilador, consulte el aviso sobre la optimización.

New analysis, similar crash. This time I launched VTune via gdb.

beasley@rbeasley-lx ~> gdb amplxe-gui
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-gui...(no debugging symbols found)...done.
(gdb) r
Starting program: /opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-gui
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe4cd3700 (LWP 29705)]
[New Thread 0x7fffdffff700 (LWP 29706)]
[New Thread 0x7fffdf1ca700 (LWP 29707)]
[New Thread 0x7fffd77e1700 (LWP 29848)]
[New Thread 0x7fffd6fe0700 (LWP 29849)]

** (amplxe-gui:29696): CRITICAL **: os_bar_hide: assertion `OS_IS_BAR (bar)' failed

(amplxe-gui:29696): Gtk-CRITICAL **: IA__gtk_widget_hide: assertion `GTK_IS_WIDGET (widget)' failed

** (amplxe-gui:29696): CRITICAL **: os_bar_set_parent: assertion `OS_IS_BAR (bar)' failed

** (amplxe-gui:29696): CRITICAL **: os_bar_hide: assertion `OS_IS_BAR (bar)' failed

(amplxe-gui:29696): Gtk-CRITICAL **: IA__gtk_widget_hide: assertion `GTK_IS_WIDGET (widget)' failed

** (amplxe-gui:29696): CRITICAL **: os_bar_set_parent: assertion `OS_IS_BAR (bar)' failed
[New Thread 0x7fffc5522700 (LWP 30043)]
[Thread 0x7fffdf1ca700 (LWP 29707) exited]
[New Thread 0x7fffdf1ca700 (LWP 30100)]
[New Thread 0x7fffbd880700 (LWP 30102)]
[New Thread 0x7fffbdc81700 (LWP 30101)]
[New Thread 0x7fffbcc7d700 (LWP 30105)]
[New Thread 0x7fffbd07e700 (LWP 30104)]
[New Thread 0x7fffbd47f700 (LWP 30103)]
[New Thread 0x7fffbc87c700 (LWP 30106)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffbcc7d700 (LWP 30105)]
0x00007fffc6524015 in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
(gdb) bt
#0 0x00007fffc6524015 in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#1 0x00007fffc651e187 in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#2 0x00007fffc651d6a5 in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#3 0x00007fffc654154a in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#4 0x00007fffc654147c in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#5 0x00007fffc6540f4a in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#6 0x00007fffc650a74f in ?? () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
#7 0x00007fffc8578448 in dicerresolver_2_2::internal::ism_module_bank::get_symbol(dicerresolver_2_2::internal::segment_rva const&) ()
from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_impl_2.2.so
#8 0x00007fffc85a943d in dicerresolver_2_2::internal::virtual_stack::virtual_stack(dicerresolver_2_2::internal::resolver_impl&, dbinterface_1_38::Index, dicerresolver_2_2::internal::segment_rva const&, boost::shared_ptr, dicerresolver_2_2::resolution_type_set&) ()
from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_impl_2.2.so
#9 0x00007fffc85945d8 in dicerresolver_2_2::internal::resolver_impl::resolve_code_locations(dicerresolver_2_2::internal::attribute_row_ref_table*, dbinterface_1_38::Index, dbinterface_1_38::Index, boost::shared_ptr, msngr2::IProgress*, dicerresolver_2_2::resolution_type_set&, dicerresolver_2_2::internal::call_target_row_ref_table&) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_impl_2.2.so
#10 0x00007fffc85a4c02 in dicerresolver_2_2::internal::resolver_impl::resolve_module(dbinterface_1_38::Index, msngr2::IProgress*, dicerresolver_2_2::internal::call_target_row_ref_table&) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_impl_2.2.so
#11 0x00007fffc82bb5ad in dicerresolver_2_2::internal::parallel_module_loading::do_resolve(msngr2::IProgress*, dbinterface_1_38::Index, dicerresolver_2_2::internal::call_target_row_ref_table&) const () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_core_2.2.so
#12 0x00007fffc82bbe41 in dicerresolver_2_2::internal::parallel_module_loading::operator()(tbb::blocked_range const&) const ()
from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_core_2.2.so
#13 0x00007fffc82d7344 in tbb::internal::start_for, dicerresolver_2_2::internal::parallel_module_loading, tbb::auto_partitioner>::execute()
() from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_dicerresolver_core_2.2.so
#14 0x00007fffd7800911 in tbb::internal::custom_scheduler::local_wait_for_all(tbb::task&, tbb::task*) ()
from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libtbb.so.2
#15 0x00007fffd77fe6fc in tbb::internal::arena::process(tbb::internal::generic_scheduler&) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libtbb.so.2
#16 0x00007fffd77fd83c in tbb::internal::market::process(rml::job&) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libtbb.so.2
#17 0x00007fffd77fb35b in tbb::internal::rml::private_worker::run() () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libtbb.so.2
#18 0x00007fffd77fb266 in tbb::internal::rml::private_worker::thread_routine(void*) () from /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libtbb.so.2
#19 0x00007ffff434be9a in start_thread (arg=0x7fffbcc7d700) at pthread_create.c:308
#20 0x00007ffff29fe3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#21 0x0000000000000000 in ?? ()
(gdb) info reg
rax 0x1 1
rbx 0x1a883 108675
rcx 0x1a883 108675
rdx 0x1a87a 108666
rsi 0x7fffa8bb7570 140736024245616
rdi 0x7fffbcc79c38 140736360586296
rbp 0x1a87a 0x1a87a
rsp 0x7fffbcc79ae0 0x7fffbcc79ae0
r8 0x1 1
r9 0x8 8
r10 0x7fffc677e368 140736523133800
r11 0x1a87a 108666
r12 0x7fffbcc79c38 140736360586296
r13 0x8 8
r14 0x7fffa8bb7570 140736024245616
r15 0x1 1
rip 0x7fffc6524015 0x7fffc6524015
eflags 0x10202 [ IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
(gdb) info share /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
From To Syms Read Shared Object Library
0x00007fffc64f3ef0 0x00007fffc6556278 Yes (*) /opt/intel/vtune_amplifier_xe_2013/lib64/../lib64/../lib64/libamplxe_ism_core_3.19.so
(*): Shared library is missing debugging information.

Thanks for this valuable report.

Did you work on advanced-hotspots data collector? To simplify the problem, try command line:

amplxe-cl -collect advanced-hotspots -no-auto-finalize -duration 60 -target-pid PID

Then, you can try:

amplxe-cl -finalize -r r00?ah ; have you seen similar problem in GUI? What is error message?

I wonder that the tool will not crash when resolving symbol info:-)

You must try to keep the amount of data collection within limits.  The sampling rate can be adjusted by command line sample-after value or by the estimated time in the Advanced section of the GUI setup.  The estimated time is self-explanatory for the case where there are perhaps up to 4 total threads running on the platform.  Beyond that, you must reduce estimated time to a fraction of real time.  The .tb6 file size limits were increased several updates back, but even 300MB may be excessive.

It seems that faulting IP is located at this code point rip 0x7fb483a16448 0x7fb483a16448 <dicerresolver_2_2::internal::ism_module_bank::get_symbol(dicerresolver_2_2::internal::segment_rva const&)+264>.

Now because this  function belongs to VTune library so the source code and signature will not be available.By looking at function description and if the error is signaled as a segfault I can theorize that probably the value passed by reference could point to non-accessible memory.It can be this value segment_rva const&.

As I do not have an experience with GDB so I would like to ask you to resolve with the help of GDB address which containes rsi register(rsi 0x7fb55deb53b0).

Zitat:

Peter Wang (Intel) schrieb:

Did you work on advanced-hotspots data collector? To simplify the problem, try command line:

amplxe-cl -collect advanced-hotspots -no-auto-finalize -duration 60 -target-pid PID

Then, you can try:

amplxe-cl -finalize -r r00?ah ; have you seen similar problem in GUI? What is error message?

Using the GUI, I've tried the Basic Hotspots, Advanced Hotspots, and General Exploration - Nehalem / Westmere analysis types. The Basic Hotspots analysis is the least flaky, but still fails for the same reason the majority of the time.

I followed your suggestions using the command line interface (amplxe-cl), and everything appears stable. The collect and finalize steps both succeeded. Next I changed into my the project folder for my application, and I reran "amplxe-cl -finalize" against each of them. The command consistently succeeded. :D

Going back to the original report about the GUI, the UI never displayed an explicit error message. It'd only trigger a SIGSEGV, which would be handled by the UI popping up a "would you like to submit this crash report to Intel?" dialog.

So, as of writing, I now have a workaround by dealing with sample collection via command line, and I'll open the results in the GUI for review. Especially considering the price attached to the package, I'd still prefer that the GUI not explode, however. ;)

Thanks for the assist!

Zitat:

Tim Prince schrieb:

You must try to keep the amount of data collection within limits.  The sampling rate can be adjusted by command line sample-after value or by the estimated time in the Advanced section of the GUI setup.  The estimated time is self-explanatory for the case where there are perhaps up to 4 total threads running on the platform.  Beyond that, you must reduce estimated time to a fraction of real time.  The .tb6 file size limits were increased several updates back, but even 300MB may be excessive.

That's a fair observation, however I should note that I'm sampling an application for less than a minute, and my sampled data (r?????/data.0) maxes out at around 25M for the longest sample duration. I don't think I was collecting excessive amounts of information. :) That said, I'd find it kinda ironic that a developer tool geared toward helping people write robust code crashes 'cuz it may have been fed too much data. (And it's not like some 3rd party program generated the input data!)

Zitat:

iliyapolak schrieb:

As I do not have an experience with GDB so I would like to ask you to resolve with the help of GDB address which containes rsi register(rsi 0x7fb55deb53b0).

gdb was unable to resolve the address to anything. :\

>>>gdb was unable to resolve the address to anything. :\>>>

Maybe this is the culprit?Btw can explain what is the error code?Is that segfault?

>>>Btw can explain what is the error code?Is that segfault?>>>

Please disregard this quoted sentence,because I have found the error description in your first post.

Regarding the mentioned  address which holds rsi register I meant possibility to resolve the address to some memory region like process heap.

Hello Ryan!

When GUI handled SIGSEGV, were you able to submit crash report? That could help us to find the reason of GUI problem.

Zitat:

Vitaly Slobodskoy (Intel) schrieb:

Hello Ryan!

When GUI handled SIGSEGV, were you able to submit crash report? That could help us to find the reason of GUI problem.

I avoided sending the crash reports, because I was concerned that doing so might leak confidential information. I'm working on another task now, but upon switching back I'll give this a shot. Thanks!

Deje un comentario

Por favor inicie sesión para agregar un comentario. ¿No es socio? Únase ya