For firmware, OS level system and device driver debug, using a JTAG interface is the most commonly used method in the embedded intelligent systems world. The same is also true in the mobile device space. This article covers the use of the Intel(R) System Debugger for system software stack debug on Android* based IA-32 and Intel(R) 64 devices.
The Joint Test Action Group (JTAG) IEEE 1149.1 standard defines a “Standard Test Access Port and Boundary-Scan Architecture for test access ports used for testing printed circuit boards.” This standard is commonly simply referred to as the JTAG debug interface. From its beginnings as a standard for circuit board testing it has developed into the de facto interface standard for OS independent and OS system level platform debug.
More background information on JTAG and its usage in modern system software stack debugging is available at in the article “JTAG 101; IEEE 1149.x and Software Debug”.
From the OEM’s perspective and that or their partner application and driver developers understanding the interaction between the driver and software stack components running on the different parts of the Systen-on-Chip (SoC) integrated intelligent system or smartphone form factor device is critical for determining platform stability, and from a silicon validator’s perspective the low level software stack provides the test environment that exposes the kind of stress factors the platform will be exposed to in real world use cases. In short, modern SoCs require understanding the complete package and its complex real-world interactions, not just positive unit test results for individual hardware components. This is the level of insight a JTAG based system software debug approach can provide. This can be achieved by merging the in-depth hardware awareness JTAG inherently provides with the ability to export state information of the Android* OS running on the target.
Especially for device driver debug, it is important to understand both the exact state of the peripheral device on the chipset and the interaction of the device driver with the OS layer and the rest of the software stack.
If you are looking at Android* from the perspective of system debugging, looking at device drivers and the OS kernel, it is really just a specialized branch of Linux*. Thus it can be treated like any 2.6.3x or higher Linux*.
The Intel® Atom™ Processor Z2xxx, Z3xxx as well as the the latest generation Intel® Core™ processor that most of the current Intel® architecture based Android* designs are based on supports IEEE-1149.1 and IEEE-1149.7 (JTAG) Boundary Scan and MPI Parallel Trace Interface (PTI) as well as Branch Trace Storage (BTS) and Last Branch Record (LBR) based instruction tracing through Intel’s JTAG compliant eXtended Debug Port (XDP).
The Intel(R) System Debugger supports XDP through the Intel(R) ITP-XDP 3BR device available for order at https://designintools.intel.com/product_p/itpxdp3brext.htm.
Each target platform must be enabled to support JTAG debugging, esp. there must be an XDP connector on the PCB such as on this board:
Consult your system vendor's documentation for details about the XDP connector and enabling of JTAG debugging.
On the host computer both Linux and Windows operating systems are supported. Detailed and up-to-date software and hardware requirements for the host and target environments are described in the Intel® System Studio 2014 release notes (links are in the reference section below) and the Intel® JTAG Debugger User and Reference Guide installed with Intel® System Studio.
What complicates debugging an Android* based platform is that Android usually very aggressively takes advantage of low power idle states and sleep states to optimize for power consumption. Thus the real challenge becomes debugging through low power states and
- either maintaining JTAG functionality through some of the low power states
- or, where this is not possible re-attaching JTAG as soon as the chipset power domain for JTAG is re-enabled.
Many OS level issues on these types of platforms tend to center around power mode changes and sleep / wake-up sequences.
A system debugger, whether debug agent based or using a JTAG device interface is a very useful tool to help satisfy several of the key objectives of OS development.
The debugger can be used to validate the boot process, analyze and correct stability issues like runtime errors, segmentation faults, or services not being started correctly during boot.
It can also be used to identify and correct OS configuration issues by providing detailed access and representations of page tables, descriptor tables and also instruction trace. The combination of instruction trace and memory table access can be a very powerful tool to identify the root causes for stack overflow, memory leak or even data abort scenarios.
Device Driver Debugging
A good System Debug solution for OS level debug should furthermore provide visibility of kernel threads and active kernel modules along with other information exported by the kernel. To allow for debugging dynamically loaded services and device drivers a kernel patch or a kernel module that exports the memory location of a driver’s initialization method and destruction method may be used. Especially for system configuration and device driver debugging, it is also important to be able to directly access and check the contents of device configuration registers.
Analyzing the code after the Android* compressed zImage kernel image has been unpacked into memory, is possible by simply releasing run control in the debugger until start_kernel is reached. This implies of course that the vmlinux file that contains the kernel symbol info has been loaded. At this point the use of software breakpoints is possible. Prior to this point in the boot process only breakpoint register based hardware breakpoints should be used, to avoid the debugger attempting to write breakpoint instructions into uninitialized memory. The operating system is then successfully booted once the idle loop mwait_idle has been reached.
Additionally, if your debug solution provides access to instruction trace, this capability can in conjunction with all the regular run control features of a system software debug help to identify and resolve runtime issues and segmentation faults.
It can also be used to identify and correct OS configuration issues by providing detailed access and representations of page tables, descriptor tables, and also instruction trace. The combination of instruction trace and memory table access can be a very powerful tool to identify the root causes for stack overflow, memory leak, or even data abort scenarios.
The figure below shows the page table translation from physical to virtual memory addresses. With the high level of flexibility that is available on x86 in defining the depth of translation tables and granularity of the addressed memory blocks, this level of easy access and visibility of the memory layout becomes even more important for system development on the OS level.
To translate a logical address into a linear address, the processor does the following:
- Uses the offset in the segment selector to locate the segment descriptor for the segment in the GDT or LDT and reads it into the processor. (This step is needed only when a new segment selector is loaded into a segment register.)
- Examines the segment descriptor to check the access rights and range of the segment to insure that the segment is accessible and that the offset is within the limits of the segment.
- Adds the base address of the segment from the segment descriptor to the offset to form a linear address.
- If paging is not used, the processor maps the linear address directly to a physical address (that is, the linear address goes out on the processor’s address bus). If the linear address space is paged, a second level of address translation is used to translate the linear address into a physical address.
When operating in protected mode, the Intel Architecture permits the linear address space to be mapped directly into a large physical memory (for example, 4 GBytes of RAM) or indirectly (using paging) into a smaller physical memory and disk storage. This latter method of mapping the linear address space is commonly referred to as virtual memory or demand-paged virtual memory.
When paging is used, the processor divides the linear address space into fixed-size pages (generally 4 KBytes in length) that can be mapped into physical memory and/or disk storage. When a program (or task) references a logical address in memory, the processor translates the address into a linear address and then uses its paging mechanism to translate the linear address into a corresponding physical address. If the page containing the linear address is not currently in physical memory, the processor generates a page-fault exception (#PF). The exception handler for the page-fault exception typically directs the operating system or executive to load the page from disk storage into physical memory (perhaps writing a different page from physical memory out to disk in the process). When the page has been loaded in physical memory, a return from the exception handler causes the instruction that generated the exception to be restarted. The information that the processor uses to map linear addresses into the physical address space and to generate page-fault exceptions (when necessary) is contained in page directories and page tables stored in memory.
Paging is different from segmentation through its use of fixed-size pages. Unlike segments, which usually are the same size as the code or data structures they hold, pages have a fixed size. If segmentation is the only form of address translation used, a data structure present in physical memory will have all of its parts in memory. If paging is used, a data structure can be partly in memory and partly in disk storage.
To minimize the number of bus cycles required for address translation, the most recently accessed page-directory and page-table entries are cached in the processor in devices called translation lookaside buffers (TLBs). The TLBs satisfy most requests for reading the current page directory and page tables without requiring a bus cycle. Extra bus cycles occur only when the TLBs do not contain a page-table entry, which typically happens when a page has not been accessed for a long time.
Especially for system configuration and device driver debugging, it is also important to be able to directly access and check the contents of device configuration registers. A bitwise visualization makes it easier to catch and understand changes to a device state during debug, while the associated device driver is interacting with it.
Analyzing the code after the Android compressed zImage kernel image has been unpacked into memory is possible by simply releasing run control in the debugger until start_kernel is reached. This implies of course that the vmlinux file that contains the kernel symbol information has been loaded. At this point the use of software breakpoints is possible. Prior to this point in the boot process only breakpoint-register–based hardware breakpoints should be used, to avoid the debugger attempting to write breakpoint instructions into uninitialized memory. The operating system is then successfully booted once the idle loop mwait_idle has been reached.
Last Branch Records can be used to trace code execution from target reset. Since discontinuities in code execution are stored in these MSRs, debuggers can reconstruct executed code by reading the ‘To’ and ‘From’ addresses, access memory between the specific locations, and disassemble the code. The disassembly is displayed in a trace GUI in the debugger interface. This is useful for seeing what code was executed before a System Management Interrupt (SMI) or other exception if a breakpoint is set on the interrupt.
Android* Specific Considerations
Device Driver Debug:
- Modules have to be signed in Android 4.x.x.
- Makefile of xdbntf needs a small change:
diff xdbntf/Makefile xdbntf.old/Makefile
< $(MAKE) -C $(ANDROID_PRODUCT_OUT)/linux/kernel M=$(PWD) modules
> $(MAKE) -C $(ANDROID_PRODUCT_OUT)/kernel_build M=$(PWD) modules
- Android environment must be setup before compiling xdbntf (basically to set the correct ANDROID_PROCUCT_OUT directory):
- cd [android_root]
- source build/envsetup.sh
- cd [xdbntf_sources}
- adb push xdbntf.ko /data
- Insmod the xdbntf.ko file
- adb shell
- insmod /data/xdbntf.ko
If the user build xdbntf.ko and tries to insmod it without the correct signature he will see an error message like: “insmod: init_module '/data/xdbntf.ko failed (Required key not available)”
- Xdbntf notifications:
Due to limitations in silicon debug features, kernel module load notifications may not function correctly on Intel(R) Atom(TM) Procerssor Z35xx platfoms.The following notes should be observed:
1 – user must set a software breakpoint in the target prior to using xdbntf. The breakpoint should be in an unused/unreachable code location, it is not necessary that the breakpoint is ever hit, its purpose is to enable the JTAG debugger redirection logic in the Silicon, which will allow the kernel module notifications to function correctly.
2 – in some cases the notification will only partially work: this will manifest as a hung system, with the JTAG debugger indicating a “running” state. In this case the user should manually halt the target, at which point the debugger will detect the notification, consume it, and resume target execution.
Intel(R) System Debugger Feature Overview
Full Intel Silicon & Chipset support
An in-depth view into Intel® processors supporting silicon specific features, including - architectural registers - Intel® Streaming SIMD Extensions - as well as graphics chipset registers for Atom™ Processor chipsets. You will find the graphical representation of peripheral registers, corresponding bit fields and their online documentation from within the debugger GUI very helpful when examining or altering a system configuration.
Execution Trace Support
View execution history and gain a deep understanding of the system wide executed code flow. Find errors by analysing the execution path and identifying the root cause of exceptions.
Updated Source Files Window
The Source Files Window shows, within a tree structure, all source files loaded for the target. At top level the libraries, below that the folders and as leaf nodes the source files itself are displayed. Use the search field to filter the displayed files. Double click on a source file in this tree to open the file in a Source Window.
Linux* OS Awareness / Kernel Module Debugging
Debugging a Linux* system is much easier with the Intel® System Debugger. It shows all currently active kernel threads and a list of all currently loaded kernel modules with status information and the memory location of the initialization and clean-up methods. To debug a kernel module set breakpoints on module load, initialization or clean-up/exit. On reaching a breakpoint the module source code is displayed - now single step through your code. Unloaded kernel modules are debugged by entering the modules name and then defining breakpoints just as with modules already loaded into the OS.
Power Events Handling
If a target is reset or powered-off the debugger retains the setup and once power is restored the debugger will attempt to halt the target at the reset vector.
It is very easy to translate any address between the physical and virtual address spaces. Use this to identify what physical memory areas your code is actually accessing or running from.
A rich set of powerful debugging commands are available for debugging in a non-interactive mode. Results can be logged and analysed on completion. This feature is ideally suited for regression testing. Here is a simple example I used after resetting the target to get over the initial start-up code and load symbols for the EFI firmware I was debugging:
- Remote Application Debug on Android* OS: http://software.intel.com/en-us/articles/application-debug-android/
- Debugging Android* OS running on Intel® Atom™ Processor via JTAG: http://software.intel.com/en-us/articles/debugging-android-atom-jtag/
- JTAG 101; IEEE 1149.x and Software Debug” by Randy Johnson and Stewart Christie: (http://download.intel.com/design/intarch/papers/321095.pdf).