Third-Party Android* Application Debug Reference on Intel® Processor-based Platforms Part 1

Contents

Third-Party Application Debug Reference on Intel® Processor-based Platforms
Introduction
Debug Tricks

Call Stack
Log Tools
Wakelock

Miscellanous Debugging Tricks MethodTracing
HProf (Heap Profile)
SamplingProfile
System Signal
Logcat
jdwp (Java debug wire protocol)
android.os.Debug

Debug Tools

GDB

Target Device Side:
Host PC Side:
gdb multi-thread debug command
Debug Case:  Debug service in system_server process
Debug Case:  Debug Android app with native library[put a lead-in sentence as to what this section is for]
Android Core dump file Analysis
TroubleShooting in Eclipse*

kprobe]

How to use kprobe kernel debug

Introduction

Developing applications is important for Intel® processor-based mobile platforms to be successful. For platform engineers and application engineers who want to enable applications as much as possible on Intel platform, there are no source code for applications from third-party ISV (e.g., Google ), there is a big question about how to debug these no source code applications on intel platform.

This document shows how the debugging experience, detailed methology, and tool usage for debugging no–source-code third-party applications on Intel processor-based platforms.

Debug Tricks

Call Stack

  • Description:
    The call stack is important for debugging because it tells you where the bug occurs in the source code. It’s a running history, if you will. There are call stacks for Java* space and native space and different ways to print them as the following paragraphs show.

  • Print Java Space Call Stack:

    1. Method that will not break the program which you are debugging.

      Import android.util.Log;
      void printJavaCallStack() {
      java.util.Map<Thread, StackTraceElement[]> ts = Thread.getAllStackTraces();   
          StackTraceElement[] ste = ts.get(Thread.currentThread());   
          for (StackTraceElement s : ste) {   
          	Log.d("zwang",s.toString());
          }
      }
      
    2. Method that will break program, so do not use.

      new RuntimeException("stack").printStackTrace();
      
  • Print Native Space Call Stack:

    1. Method that will not break the program which you are debugging.

      include <utils callstack="" h="">
      using namespace android;
      namespace android {
      	void get_backtrace()
      	{
      		CallStack stack;
      		stack.update();
      		stack.dump("");
      	}
      };
      
    2. Method that will break the program, so do not use until it if necessary

      int* p = NULL;
      *p = 0x8888;
      
  • Print Stack from Native Space to Java Space

    1. Apply patch 0001-Dalvik-add-support-of-print-Java-Stack-from-Native-s.patch into Dalvik project.

    2. Make Dalvik project and push libdvm.so into /system/lib on the device.

    3. After reboot, you can use Dalvik’s interface in two ways to dump the stack from native space to Java space of the process into the /sdcard/logs/ javastack file

      By shell command:

      kill -31 <pid>
      

      By API Interface:

      Add sentence “kill(getpid(),31);” at that point in the source code where you want to dump the stack from native space to java space

      For example:

      <JB>/frameworks/native/libs/binder/IServiceManager.cpp
          virtual sp<IBinder> getService(const String16& name) const
          {
              kill(getpid(),31);
             …
           }
      

      Check the Java stack in /sdcard/logs/ javastack on the device. You can find the whole call stack from native space to java space, then you will know what java function and native library are called.

      root@android:/sdcard/logs # cat javastack                                      
      
      ----- pid 25653 at 1982-01-01 02:15:14 -----
      Cmd line: com.android.providers.calendar
      
      DALVIK THREADS:
      (mutexes: tll=0 tsl=0 tscl=0 ghl=0)
      
      "main" prio=5 tid=1 NATIVE
        | group="main" sCount=0 dsCount=0 obj=0x417c2550 self=0x417b2af0
        | sysTid=25653 nice=0 sched=0/0 cgrp=apps handle=1074057536
        | schedstat=( 13633356 12645753 23 ) utm=0 stm=1 core=1
        #00  pc 000b01ad  /system/lib/libdvm.so
        #01  pc 000907ee  /system/lib/libdvm.so
        #02  pc 00091ad4  /system/lib/libdvm.so
        #03  pc 0008a33d  /system/lib/libdvm.so
        #04  pc 00000400  [vdso]
        at android.view.Display.init(Native Method)
        at android.view.Display.<init>(Display.java:57)
        at android.view.WindowManagerImpl.getDefaultDisplay(WindowManagerImpl.java:630)
        at android.app.ActivityThread.getDisplayMetricsLocked(ActivityThread.java:1530)
        at android.app.ActivityThread.applyConfigurationToResourcesLocked(ActivityThread.java:3649)
        at android.app.ActivityThread.handleBindApplication(ActivityThread.java:3969)
        at android.app.ActivityThread.access$1300(ActivityThread.java:130)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1255)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:137)
        at android.app.ActivityThread.main(ActivityThread.java:4745)
        at java.lang.reflect.Method.invokeNative(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:511)
        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:786)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:553)
        at dalvik.system.NativeStart.main(Native Method)
      
    4. patch 0001-systemcore-add-Dalvik-Tombstone-call-stack-support.patch for system/core project is optional , it just add tomestone with print java stack into /sdcard/logs/ javastack.

      0001-Dalvik-add-support-of-print-Java-Stack-from-Native-s.patch 0001-systemcore-add-Dalvik-Tombstone-call-stack-support.patch

Log Tools

logcat - Android* log message application

  • Process interface:

    cat /proc/kmsg – kernel debug message
    /proc/cpuinfo
    /proc/meminfo
    /proc/iomem
    /proc/vmallocinfo
    /proc/interrupts
    /proc/sys/vm/drop_caches
    
  • Information dump:

    procrank: process memory rank
    procmem: a specific process’ memory
    showslab: kernel slab utilization, /proc/slabinfo
    latencytop: CONFIG_LATENCYTOP
    showmap: process memory mmap address space; /proc/XXX/maps
    dumpstate- system information like memory , cpu etc
    dumpsys – system service information etc
    to see all of the "subcommands" of dumpsys do:
    dumpsys | grep DUMP
    
    DUMP OF SERVICE SurfaceFlinger:
    DUMP OF SERVICE accessibility:
    DUMP OF SERVICE account:
    DUMP OF SERVICE activity:
    DUMP OF SERVICE alarm:
    DUMP OF SERVICE appwidget:
    DUMP OF SERVICE audio:
    DUMP OF SERVICE backup:
    DUMP OF SERVICE battery:
    DUMP OF SERVICE batteryinfo:
    DUMP OF SERVICE clipboard:
    DUMP OF SERVICE connectivity:
    DUMP OF SERVICE content:
    DUMP OF SERVICE cpuinfo:
    DUMP OF SERVICE device_policy:
    DUMP OF SERVICE devicestoragemonitor:
    DUMP OF SERVICE diskstats:
    DUMP OF SERVICE dropbox:
    DUMP OF SERVICE entropy:
    DUMP OF SERVICE hardware:
    DUMP OF SERVICE input_method:
    DUMP OF SERVICE iphonesubinfo:
    DUMP OF SERVICE isms:
    DUMP OF SERVICE location:
    DUMP OF SERVICE media.audio_flinger:
    DUMP OF SERVICE media.audio_policy:
    DUMP OF SERVICE media.player:
    DUMP OF SERVICE meminfo:
    DUMP OF SERVICE mount:
    DUMP OF SERVICE netstat:
    DUMP OF SERVICE network_management:
    DUMP OF SERVICE notification:
    DUMP OF SERVICE package:
    Permission [android.permission.DUMP] (49f43060):
    perm=Permission{49fc39e0 android.permission.DUMP}
    android.permission.DUMP
    DUMP OF SERVICE permission:
    DUMP OF SERVICE phone:
    DUMP OF SERVICE power:
    DUMP OF SERVICE reboot:
    DUMP OF SERVICE screenshot:
    DUMP OF SERVICE search:
    DUMP OF SERVICE sensor:
    DUMP OF SERVICE simphonebook:
    DUMP OF SERVICE statusbar:
    DUMP OF SERVICE telephony.registry:
    DUMP OF SERVICE throttle:
    DUMP OF SERVICE usagestats:
    DUMP OF SERVICE vibrator:
    DUMP OF SERVICE wallpaper:
    DUMP OF SERVICE wifi:
    DUMP OF SERVICE window:
    
    dumptcp – tcp/ip information
    bugreport 
    

Wakelock

  • Description:
    A locked wakelock, depending on its type, prevents the system from entering suspended or other low-power states. When creating a wakelock, you can select its type. If the type is set to WAKE_LOCK_SUSPEND, the wakelock prevents a full system suspend. If the type is WAKE_LOCK_IDLE, low-power states that cause large interrupt latencies, or that disable a set of interrupts, will not be entered from idle until the wakelocks are released. Unless the type is specified, this document refers to wakelocks with the type set to WAKE_LOCK_SUSPEND.

    If the suspend operation has already started when locking a wakelock, it will abort the suspend operation as long it has not already reached the suspend_late stage. This means that locking a wakelock from an interrupt handler or a freezeable thread always works, but if you lock a wakelock from a suspend_late handler you must also return an error from that handler to abort suspend.

  • Debug Method:
    To check the wakelock status, use cat /proc/wakelocks
    name – the component that holds wakelock
    wake_count – the count of holding wakelock
    active_since – the time interval from the last time holding the wakelock

  • Tools:
    CPUSpy.apk –Use this application to get the device’s deep sleep time and to find out whether the device has a problem going into deep sleep.
    get_activewakelock.sh – Use this script to get the name and active_since columns from /proc/wakelocks
    Both CPUSpy.apk and get_activewakelock.sh are attached as following:

    get_activewakelock.sh CPUSpy.apk

Miscellanous Debugging Tricks MethodTracing

Use MethodTracing to find hot spots and analyze performance. You can also check CPU usage, function call times, etc.

Follow these steps to do a trace:

import android.os.Debug;     
……     
android.os.Debug.startMethodTracing(“/data/tmp/test”); // create /data/tmp     
…… // the program to be trace here    
 android.os.Debug.stopMethodTracing();  

after running, there will be trace file in /data/tmp/test.trace

copy trace file into PC host.

$ adb pull /data/tmp/test.trace ./

use trace tool in Android SDK for trace analysis.

$ $ANDROID_SRC/out/host/linux-x86/bin/traceview test.trace
$ANDROID_SRC/out/host/linux-x86/bin/dmtracedump -g test.png test.trace

Note:
There is a conflict between trace creation and the DEBUG version libdvm. Use the trace method only for the non-DEBUG version build.

HProf (Heap Profile)

Use HProf to analyze Java memory , show Dalvik memory usage , memory leakage , etc.

Follow these steps to do a heap profile:

import android.os.Debug;    
import java.io.IOException;     
……     
 try {    
android.os.Debug.dumpHprofData(“/data/tmp/input.hprof”); // create /data/tmp
 } catch (IOException ioe) {     
 } 

copy hprof file into PC host

$ adb pull /data/tmp/input.hprof ./

use hprof-conv to turn hprof into format by MAT tool

$ $ANDROID_SRC/out/host/linux-x86/bin/hprof-conv input.hprof output.hprof

use MAT to open hprof file to check result

MAT link: http://www.eclipse.org/mat/downloads.php

Note:

The tool only shows Java spaceb, not native space, memory usage.

SamplingProfile

do sample at millisecond interval for routine, then output sample log.

Follow these steps to do sample profile

import dalvik.system.SamplingProfiler     
……    
SamplingProfile sp = SamplingProfiler.getInstance();     
sp.start(n); // n is sample times   
 sp.logSnapshot(sp.snapshot());     
 ……     
sp.shutDown(); 
//there will be a sample thread to output information in logcat

System Signal

Use this tool to send system signal SIGQUIT and SIGUSR1 to Dalvik, which will handle these signals (dalvik/vm/SignalCatcher.c) to print the call stack or memory usage.

Follow these stpes to send system signal and get call stack

  1. $ chmod 777 /data/anr -R 
    $ rm /data/anr/traces.txt 
    $ ps # find pid
    $ kill -3 pid # send SIGQUIT to process to get tracefile
    $ cat /data/anr/traces.txt 
    
  2. $ chmod 777 /data/misc -R 
    $ ps # find pid 
    $ kill -10 pid # send SIGQUIT to process to get hproffile
    $ ls /data/misc/*.hprof 
    

Logcat

Use this tool to get aplog print from android system.

You can use following methods to add aplog into or get aglog.

  1. android.util.Log uses println for Jjava output with I/V/D….

  2. Dalvik uses pipe and thread., Use dup2 to make stdoutand stderr re-direction to pipe (vm/StdioConverter.c:dvmstdioConverterStartup), start a thread to read pipe (dalvik/vm/StdioConverter.c:stdioconverterThreadStart()), then use the LOG tool to output the log into(system/core/liblog/logd_write.c: __android_log_print())/dev/log/*

  3. The parameters for the logcat tool are:

    # logcat -b main //show main buffer
    # logcat -b radio //show radio buffer
    # logcat -b events //show event buffer
    

jdwp (Java debug wire protocol)

The Java Debug Wire Protocol (JDWP) is the protocol used for communication between a debugger and the Java virtual machine (VM) which it debugs . In Android system, JDWP is proocal used between adb and java application on android device. Developer can use it for many debug proposal.

Go to this link for more information:

http://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdwp-spec.html

android.os.Debug

Android’s Debug tool, android.os.Debug, has many debug APIs. Go to this link for more information:

More information can be found in link:

http://developer.android.com/reference/android/os/Debug.html

  1. Get nano level time

    threadCpuTimeNanos()
    
  2. Get memory allocation

    startAllocCounting()   
    stopAllocCounting()   
    getGlobalAllocCount()   
    get….. 
    
  3. Print the classes loaded in current process

    getLoadedClassCount() 
    printLoadedClasses() //it needs to open NDEBUG function
    

Debug Tools

Powerful debug tools help developer to root cause issue quickly and easily . This chapter will introduce typical android debug tools and technique about how to use them to root cause issues.

GDB

Print log is one way to debug Android apps, but it is inefficient and difficult to use.

Gdb is a good tool for debugging in single step and looking directly into source code issues. This section explains how to use the gdb tool on Android platforms.

Target Device Side:

gdbserver :<port> –attach <PID>

Host PC Side:

  1. adb forward tcp:<port> tcp:<port>
  2. cd <your code base root directory> , so gdb can find the source code in the current work path.
  3. Run the command:gdb <program to debug> (the program should first be compiled with -g switch).
  4. Start debugging the program.
  5. Setup library patch with gdb command using these two commands:
    • #set solib-absolute-prefix < path of symbols> (be careful to not have a special sign in the patch ex: ~)
    • #set solib-search-path < path of lib under symbols >

    To connect to the gdbserver on the target side, run <gdb> target remot :<port>. )

Note regarding the program/library with the debug symbol:Although defaultly android build system use “-g” switch to build native library with debug symbols, it strips the debug symbols at the last build stage. So to use native library with debug symbols, you need to use the one in the “out/target/product/symbols” directory.

gdb multi-thread debug command

Gdb tool also provide commands to debug multi-thread in one process, use the following commands to do:

info threads – print all thread information for the program you are debugging

thread <tid> – switch to debugging this thread with the specified ID.

break <file name>:<line> – set break point in source code file at the specified line. This command is very useful for system_servers that have many threads.

For example, following command will set a bread point in InputDispatcher thread of system_servers process

break InputDispatcher.cpp:1280, then continue,

To debug step by step, touch the screen at the point where you want, gdb will to stop the InputDispatcher thread.

set scheduler-locking off|on|step - When you debug mult-threads, you will find many other threads are running at the same time. To debug the current thread, use the “step”/”continue” command. By using “set scheduler-locking”, you can make your current debug thread the only running thread.

off – do not lock any thread, all threads are running, which is the default.

on – only the current debug thread is running.

step – doing debug step by step, except using “next” command, Only the current debug thread is running.

Debug Case: Debug service in system_server process

This debug case shows how to debug service thread in system_server process by Gdb tool

On the target device, type

adb shell
ps | grep system_server  (ex: system_server PID is 312)
gdbserver :1234 –attach 312

On the host PC, type:

adb forward tcp:1234 tcp: 1234
cd  ~/ics
gdb out/target/product/mfld_pr2/symbols/system/bin/app_process
#set solib-absolute-prefix /home/zwang/ics/out/target/product/mfld_pr2/symbols
#set solib-search-path /home/zwang/ics/out/target/product/mfld_pr2/symbols/system/lib
#target remot :1234

Gdb will load symbols from libraries

# break InputDispatcher.cpp:1280
# continue

Touch the screen at the point where you want gdb to stop the InputDispatcher thread so you can debug by steps.

Debug Case: Debug Android app with native library [put a lead-in sentence as to what this section is for]

This case shows how to debug android application with native library by Gdb tool

On the target device, type

adb shell
ps | grep zwang.test.app  ( ex: the app PID is 123)
gdbserver :1234 –attach 123

On the host PC, type:

adb forward tcp:1234 tcp: 1234
cd  ~/ics
gdb out/target/product/mfld_pr2/symbols/system/bin/app_process
#set solib-absolute-prefix /home/zwang/ics/out/target/product/mfld_pr2/symbols
#set solib-search-path /home/zwang/ics/out/target/product/mfld_pr2/symbols/system/lib:/home/zwang/app/obj/local/x86

When you build a native library with ndk-build, the library with debug symbols is located in the obj directory. (library under lib directory is striped out of debug symbols ). In our case, it is /home/zwang/app/obj/local/x86, so we need to add this path into the library search path of gdb solib-search-path.

#target remote :1234 to connect with gdbserver

#break zwangjni_test_app.c : 12 set break point, you will get a message like “No source file named zwangjni_test_app.c, Make breakpoint pending on future shared library load?“ Y

#continue

After stopping at the break point in the native library,you can debug by steps.

Android Core dump file Analysis

When there are program exceptions, a core dump file will be created and located in /mnt/sdcard/data/logs/crashlogxx/xxxx_xxx_xxx.core. Use adb pull to copy the core dump file to the host PC.

To load the core dump file, run command: gdb <ics>/out/target/product/mfld_pr2/symbols/system/bin/app_process xxxx_xxx_xxx.core  

Set symbols path

#set solib-absolute-prefix /home/zwang/ics/out/target/product/mfld_pr2/symbols

#set solib-search-path /home/zwang/ics/out/target/product/mfld_pr2/symbols/system/lib

Then you can use commands like bt, frame, up, down, and print to check the call stack when the program has exceptions.

TroubleShooting in Eclipse*

Eclipse is an useful integrated development environment tool for Android application development. Sometimes you will meet some strange error when using it, this section will show some typical problem and tell you how to resolve it.

When you use main menu Eclipse->preferences->Android, if you see “bad version number in .class file” error message. It is due to the Eclipse environment variable having the wrong Java run -time version number.

Go to Help->About->Installation Details, to check the Eclipse environment variable and set it correctly.

kprobe]

kprobe is the Linux* kernel debug tool, it can provide developer with printing kernel level debug log.

How to use kprobe kernel debug

Follow the below steps to print kernel level log to dmesg buffer pool.

  1. Copy kprobes sample code into the Intel driver directory for building kernel module

    cd ~/aosp/hardware/intel/linux-2.6/drivers/misc
    cp –r /AOSP/hardware/intel/linux-2.6/samples/kprobes
    
  2. Change the makefile to build the kprobe sample kernel module by typing the lines below in red text.

    wang@~/r4_1_stable/hardware/intel/linux-2.6/drivers/misc >git diff
    diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
    index 166a42e..6ef0f1d 100755
    --- a/drivers/misc/Makefile
    +++ b/drivers/misc/Makefile
    @@ -3,6 +3,7 @@
     #
     
     intel_fabric_logging-objs  := intel_fw_logging.o intel_fabricerr_status.o
    +obj-m  += kprobes/
     obj-$(CONFIG_IBM_ASM)          += ibmasm/
     obj-$(CONFIG_AD525X_DPOT)      += ad525x_dpot.o
     obj-$(CONFIG_AD525X_DPOT_I2C)  += ad525x_dpot-i2c.o
    diff --git a/samples/kprobes/Makefile b/samples/kprobes/Makefile
    index 68739bc..8f253fc 100644
    --- a/samples/kprobes/Makefile
    +++ b/samples/kprobes/Makefile
    @@ -1,5 +1,8 @@
     # builds the kprobes example kernel modules;
     # then to use one (as root):  insmod <module_name.ko>
     
    +CONFIG_SAMPLE_KPROBES=m
    +CONFIG_SAMPLE_KRETPROBES=m
    +
     obj-$(CONFIG_SAMPLE_KPROBES) += kprobe_example.o jprobe_example.o
     	obj-$(CONFIG_SAMPLE_KRETPROBES) += kretprobe_example.o
    
  3. Make bootimage to build the kprobe sample kernel module, then you can find it in:

    out/target/product/mfld_pr2/kernel_build/drivers/misc/kprobes/kretprobe_example.ko
    out/target/product/mfld_pr2/kernel_build/drivers/misc/kprobes/kprobe_example.ko
    out/target/product/mfld_pr2/kernel_build/drivers/misc/kprobes/jprobe_example.ko
    
  4. re-flash phone images including boot.bin and system.img to make magic number consist between boot.bin and kprobe modules, otherwise you will fail to insert kprobe modules into kernel.

  5. To find a kprobe kernel message in /proc/kmsg, type insmod kprobe_example.ko.