The Intel Application Debugger has been tested against MeeGo* 1.0 Netbook Preview 20100330 and 20100420.
The Intel JTAG Debugger has been tested against MeeGo* 1.0 Preview 20100330 and 20100420 releases.

A key difference between embedded software development and standard desktop/laptop personal computer development is the frequent necessity to use cross-development methodology. Frequently the target system, the platform on which an application of OS software stack is intended to ultimately run, does not and is not supposed to have any development tools on it. Frequently its software stack and even the underlying platform architecture is different enough that code written for the development host would not run on the target device without significant modifications.

This implies that the code intended to run on an embedded target will have to be built either inside a protective isolating wrapper (a jailroot or chroot based environment), or at least that the code would have to be built using a build option that forces the compiler and linker to pull in the target OS-specific libraries instead of the system libraries from the development system.

Although when developing for the Intel Atom processor, the microarchitecture level differences between host and target system are probably only minimal, it is still advisable to ensure that there is no cross-system software stack pollution and try to keep them strictly separated.

The Intel® Software Development Tool Suites support a wide variety of Intel Atom processor-based devices. This means that there is not one single use case or usage model that will fit the needs of every developer. Development for Intel Atom processor-based devices tends to have one aspect in common, however. In most cases you will want to take advantage of the performance and the high screen resolution of your regular development environment and deploy your build to the real hardware for validation and analysis only. In short, most commonly the usage model will be one of cross development.

The Intel Software Development Tool Suites for Intel Atom processor address this in multiple ways.

The Intel® C++ Compiler and the Intel® Integrated Performance Primitives (Intel® IPP) will by default be installed on the development host system. However, to provide a protected build environment without host system library pollution, it is also supported to install these components into the MeeGo* Image Creator jailroot system or to install them into a developer image running inside a KVM* virtual machine or even on a real target. For the installation into MeeGo* Image Creator  or into a KVM* image, it is recommended to do this following the yum rpm repository based installation process outlined in the Moblin* Integration Guide Moblin_Integration.htm that is part of the installation image.
MeeGo* Image Creator kickstart scripts can be found at /en-us/articles/meego-integration-software-development-tool-suite-atom.
A template rpm repository definition file for the direct KVM* image installation of the Intel C++ Compiler and Intel IPP can be found at /en-us/articles/installing-the-intel-c-compiler-and-intel-integrated-performance-primitives-into-qemu-for.

The Intel® Application Debugger will with the standard installation package be installed on your development host. Via TCP/IP you can connect to and debug a process either running on actual Intel Atom processor-based target hardware, or a KVM* virtual machine, or some other virtualization method that assures you that the application to be debugged runs in a software environment identical to the later application deployment. Please also have a look at the Getting Started Guide Getting_Started.html that is part of the tool suite distribution for further details on the Intel Application Debugger usage.
The Intel JTAG Debugger will be installed on your development host with the standard installation package. Via a JTAG device, you can connect to your target platform's eXtended Debug Port (XDP) and thus commence remote cross-debugging of your entire platform and system level software.
The Intel VTune Performance Analyzer analyzes event-based and time-based sampling data on the development host. The data has been gathered on an Intel Atom processor-based target platform with low overhead, giving you accurate sampling data. The analysis can be done within the standard KDE* or Gnome* based standard Linux* GUI you are used to for your development efforts.


Cross-Debug :

If you are developing on an Intel® Core processor, for example, and both your host and target environment are Linux*-based, you may be able to get away with just using a GUI simulation layer like Xephyr*. In this case, you could at least in theory debug locally inside your chroot environment with a traditional native GDB setup. It may still be more convenient to have the rich full GUI of the Intel Application Debugger, though.

Most of the time, however, it is best to either remotely connect to real target hardware or to connect via TCP/IP to a virtual machine that runs the target environment software stack you are developing for. This way you have your target environment cleanly isolated and do not need to worry about any cross-pollution. For hardware platform optimization, it will of course be necessary to have real hardware at hand, so the baseline data and measured performance and power consumption improvements do indeed reflect the user experience on the real target.

In short, although it may be tempting when developing for and debugging application for the Intel Atom processor to take shortcuts and pretend you are doing native development, it is best to stick to the cross-development paradigms common for cases where host system and target system differ more.

Let us now look at some of the details in the debug setup when doing cross-debugging. First you will need to have the source of the code base you want to debug accessible on the debug host. In addition you will also need the symbol information. With Linux targeted development, the Dwarf symbol info is included in the linker output file. If you want the executable run on the target device to not include any symbol information, you have to strip the symbol information out before copying the file to the target device or target virtual machine.

The source directory layout stored in the symbol information (i.e., as seen by the build environment at build time), may differ from the layout the debugger sees when you want to debug your code. That may be caused by you actually having built your application on a different system than the one you are using for debugging. It may also be caused by something as simple as your built environment being inside a chroot wrapper or inside a cygwin type environment, whereas the debugger you use may be launched outside such an environment.

Debugging Applications:

When debugging applications, the solution that the Intel Software Development Tool Suites for Intel Atom Processor offers is one of a TCP/IP-based cross debug solution.

The Intel Application Debugger requires a server application running on the target system to handle communication between development host and debug target.  This program can be found at /opt/intel/atom/idb/2.x.xxx/server in the tool suite installation and simply copied over onto the target's file system. The location of the remote server on the target system is not important however it must be started before you can continue with the debug session. We recommend that you start the remote server as root to avoid any access problems if you would like to upload an application to debug. You also need to know the TCP/IP address of the target system.

When you start the remote server on the target system (whether it may be an actual hardware target or a MeeGo* developer image running inside a KVM* virtual machine) it will respond with a sign-on message on the console of the target system.

You are now ready to start the Application Debugger on the host system. Look for the idb.sh file in the /opt/intel/atom/idb/x.x.xxx/bin directory. You can start the Application Debugger with the following command:

The Application Debugger will open up for the first time with an empty window with a menu line with most buttons grayed out and below a Console window. If it has been used before, it opens with the same window layout as when it was closed. Only the Console window will contain any data - the sign in message.

Below the Console window you find a status bar. In the middle you have an indication of what mode of operation the debugger is in.

In the menu you have a few symbols active. One of them is the ‘Connect ..' button. Click on it to connect to the remote server on the target. You will see the Open Connection dialog where you can enter the IP address of the target system. You can also debug an application running on your host system. In this case you still need to start the remote server (on the host system) and connect to it via the IP address illustrated in the screen shot below:

The port number should be the same as the one used when the remote server was started. It is possible to have more than one Application Debugger running; each would be connected to a remote server with different port number. This allows you for example to debug an application both on the host and target at the same time.

On the host system, the debugger GUI will be updated to illustrate that you now are ready to attach to a running process or load a new application.

If you decide to load an application, a popup window will appear where you can browse for the application binaries, specify any specific environment variables you need, and specify where the source files are (if in a different directory than the binary):

You need to check the ‘Upload' box if you want the debugger to copy the file over to the target system. In the ‘Target Executable File' field, you enter the location on the target where you will store the executable. Make sure that all directories you specify already exist on the target system and that the file name is the same as specified in the ‘Executable File' field. (The location on the host and target can be different but the file name should be the same). When you press the OK button it will take some time after which the status indicator will change to yellow. You are now ready to start debugging.

You can set a breakpoint by just double-clicking on any of the blue dots to the left of the line numbers in the Source window. Note that if you select line 8 (in the screen shot above) you are outside of the stack frame from main and hence have no access to local variables.

Another very useful window is the Callstack, where you can see the path through the function calls you have taken to reach your current position in the code. You can open the Callstack window by selecting View/Callstack from the debugger menu or typing Ctrl+Alt+R:

Additional windows which might be useful are the Register, Locals and Threads windows. The Register window is opened with View/Register or Ctrl+Alt+G and shows the current values of the segment registers, general purpose registers, floating point registers and the xmm registers. The Locals window shows the local variables and is opened with View/Locals or Ctrl+Alt+V. The Thread window is particularly useful if your application is multithreaded. It shows which thread you are currently debugging. You open this window with View/Threads or Ctrl+Alt+T.

In short, once you accept the basic paradigm of cross-development and cross-debugging as it is commonly used in the embedded world for your Intel Atom processor targeted development work as well, developing is just as seamless and straightforward has you have always been used to.

System-Level Debug: 

For system-level debug the Intel Embedded Software Development Tool Suite for Intel Atom processor offers a fully Eclipse RCP GUI-driven debug solution taking advantage of Intel's eXtended Debug Port JTAG interface. System-level debug refers to the software development and debug that takes place on all the software stack layers between pure bare-metal development and development on the device driver and kernel module or shared object level of a complex full featured operating systems (FFOS). For our purposes, let us refer to it as progressively debugging the software stack from the firmware level upwards to the device driver level. Ever since the early 1990s the IEEE Standard 1149.x known as JTAG became the de facto standard for hardware-assisted debugging bypassing any software stack based I/O implementation. This has the advantage of providing full platform and run-time control while at the same time being flexible and affordable enough to be used for low-level and mid-level software development. This along with hardware platform validation is where JTAG is most commonly used.

Our Intel JTAG Debugger is such a cross-debug solution for system level debug.  



It provides not only a rich Eclipse RCP-based GUI and full run-control via single-step and breakpoints. It also has features such as descriptor table and page translation table access to better understand and manipulate your platform memory configuration. It adds on-chip instruction trace support for easier and more convenient analysis of run-time issues such as segmentation faults. It adds on the fly assembly code modification in RAM. And last but not least it provides bitfield editors.


The bitfield editors grant you access to all the peripheral registers and processor registers on the platform. The display them in a bitfield where each bit can be modified and toggled individually. We also provide a short description of the function of each of those registers and their bits in form of online help that is displayed below the register contents itself.

This capability can come in really handy especially when debugging device drivers and trying to understand the device register configuration changes for a specific device while they occur.

JTAG: 

Before we go into the actual debug process, let us briefly digress and talk a bit more about JTAG itself.

The Joint Test Action Group (JTAG) IEEE 1149.1 standard defines a "Standard Test Access Port and Boundary-Scan Architecture for test access ports used for testing printed circuit boards". This standard is commonly simply referred to as JTAG debug interface. From its beginnings as a standard for circuit board testing it has developed into the de-facto interface standard for OS independent and OS system level platform debug. Fig. 10 is a picture of a JTAG debug probe connected to an Intel processor.

When used as a debugging tool, it enables a developer to access an on-chip debug module, which is integrated into the CPU via the JTAG interface.JTAG revolutionized low level platform debug capabilities for the embedded software development community providing a direct serial link from the target processor to the host development machine

JTAG debugging has traditionally been used for hardware bring-up and more recently to expand the capabilities of agent-based debugging. However, JTAG based on-chip debug capabilities play a more significant role debugging SoCs. There the help to debug the operating system and middleware by isolating complex interactions between the software running on the SoC used for development such as in the mobile handset and consumer electronics devices.

With the Intel® JTAG Debugger included in the Intel Embedded Software Development Tool Suite for Intel Atom processor, we support two JTAG probes to interface with the eXtended Debug Port on Intel platforms.

On a very limited and restricted loaner basis, we support the Intel® ITP-XDP3 device for debugging on Intel® Atom™ Processor Z5xx, Z6xx, CE4100 and Intel® Media Processor CE3100 based platforms.

For Intel® Atom™ processor Z5xx and Z6xx-based platforms we also support the Macraigor Systems* usb2Demon device.



Both interfaces use a USB 2 connector on the debug host side and use drivers that are being downloaded from the host side onto the JTAG device to handle the debug communication.

To enable support for the Macraigor Systems* usb2Demon* device for debugging Intel Atom processor Z5xx or Z6xx-based platforms with the Intel JTAG Debugger, it is necessary to install the Linux* drivers for the Macraigor Systems* usb2Demon* device. The driver can be found at http://www.macraigor.com/full_gnu.htm. Please scroll down all the way and download and install the Linux32 OCDRemote package recommended at http://www.macraigor.com/intel/ . Follow the installation instructions posted at the same location closely.

1. Download mcgr-hwsupport-x.x-x.i386.rpm from http://www.macraigor.com/intel/. You will find the latest recommended version of the OCDRemote* driver set referenced there.

Support for the Intel® Atom™ processor Z5xx will require at least version 7.9.0. Support for the Intel® Atom™ processor Z6xx will require at least version 8.3.0.

1. Ensure you have root or sudo access rights on your Linux* installation.
2. Install mcgr-hwsupport-x.x-x.i386.rpm using the rpm -i command.
3. You should now be able to to launch the Intel JTAG Debugger using the xdb_Z500_mcrg.sh or xdb_Z600_mcrg.sh startup script respectively. These scripts are being put into the /opt/intel/atom/xdb/2.2.xxx/bin directory by the tool suite installer as described in the general installation section above.

Firmware & Bootloader :

The first step after the OEM or OSV receives a reference platform and is customizing it would typically be to develop and/or modify the OS bootloader. To debug a bootloader application, which runs on the target device even before any OS is present the debugger needs to be able to first of all download this relatively simple, but usually highly footprint optimized bare-metal application into pre-initialized target RAM for testing.

The debugger needs to be able to map and associate the symbol info and sources for statically linked code and to also download the application directly into target RAM for easy debug ability and testing. Once testing, developing and debugging is complete the bootloader would then go into target solid state memory.  This is usually flash, where it can do its job for preparing of target initialization and of loading the OS image from NAND,  unpacking it into RAM and launching the OS.

All these steps of bootloader development are actively supported by the Intel JTAG Debugger. To debug bootloader code, you would launch the debugger, load the Linux kernel symbol info and bootloader symbol info, and than launch the OS. Although it is a multi-step process, it is really quite straightforward:

First you connect the Intel® Debugger to the Intel Atom processor-based platform using the Intel® ITP-XDP connector.

You can do so by issuing the command xdb.sh from your Linux host command prompt in the debugger install directory (usually /opt/intel/xdb/xx.xxx/bin)

You then issue the command set opt /hard=on from the debugger command prompt to force the debugger to only use hardware breakpoints and to not try and write software breakpoint instructions into memory.

Once you do this, you can proceed setting a breakpoint in your boot loader or at OS kernel unpack and debug this part of the early platform boot process.

Once you reach the bootloader, you can single step and follow the execution flow as you please.

For high-level language symbolic debugging, you will need to have the boot loader and/or OS source tree and build environment present on your host system. To load the symbol info you would point to the ELF Dwarf file created by the linker. You can load it using the load menu and navigating to the Elf Dwarf two file in a file selection dialog.

Alternatively you can also load it from debugger command console issuing the command

LOAD /SEGMENT /DEBUG /NOLOAD /GLOBAL OF "/directory/elf-file"

Flashing:

Once testing for firmware and/or bootloader is complete, the next step may be put put it into BIOS boot ROM or NAND flash.

The Flash Memory Tool integrated into the Intel JTAG Debugger supports writing of OS Images to the unprotected NAND flash partition for Intel Atom processor Z6xx-based platforms.



When choosing an OS Image as data file in the Flash Memory Tool dialog, it automatically analyzes the file and writes the OS and OS headers to the correct location in the NAND partition.

1. To use the Flash Memory Tool  for the protected NAND partition, the platform must be booted into the firmware ‘shell':
2. Connect a serial cable to the IA SPI0 card
3. Setup HyperTerminal to the following COM settings: 115200 bits per second, 8 data bits, no parity, 1 stop bit and no flow control
4. Power on the target - Firmware Development Kit (FDK) message and shell prompt should be displayed in the HyperTerminal screen. If the firmware doesn't stop at the shell prompt, please refer to your Software Developer Platform (SDP) User Guide:
   1. VV platform: Jumper setting J1B1, pins 1-3 are BIOS Option 1 -Diagnostic Mode - enter shell by default
   2. CDK/KDK platform: SW1DB6 - SW1 DIAGNOSTIC ON - OFF = ENABLED: Enter FDK shell
5. Start the JTAG debugger and open the Flash Memory Tool dialog

There is fully GUI assisted as well as command line driven access to the flashing feature available. It allows you to flash your bootloader in hex or binary formats.

It also allows you to individually manipulate blocks, and - if you want to preserve and be able to recreate an existing image that you are about to overwrite - you can even back up flash contents into a binary file on the host.

On Intel Atom processor Z5xx-based platforms, the Flash Memory Tool in the Intel JTAG Debugger supports downloading and burning of images into target-device FLASH memory (BIOS / Firmware) on the 1MB NOR flash card on the Intel® Atom™ processor Z500 reference board. The currently supported flash types are 82802AC and M50FW080.

Below is an example of the debugger script language that could be used to automate BIOS flashing and interface with the flash programmer GUI add-on:

Flash "select /board= 'targetboard'"
Flash "change offset 0x00"
Flash "change datafile '/home/qa/BIOS50.bin'"
Flash "burn flash false true true"

The same flashing support is also available in form of the command line XFLASH - Intel JTAG Debugger Flash Memory Tool located at /opt/intel/atom/xdb/2.2.xxx/xflash-bin. Please refer to the ./xflash.sh -help option for details about the command line syntax.

OS Kernel:

The benefits of the debugger in the arena of MeeGo* and Linux* OS kernel development are that the debugger becomes a very useful development tool to help satisfy several of the key objectives of OS development.

It can be used to validate the boot process, analyze and correct stability issues like runtime errors, segmentation faults, or services not being started correctly during boot.

It can be used to identify and correct OS configuration issues by providing detailed access and representations of page tables, descriptor tables and also instruction trace. The combination of instruction trace and memory table access can be a very powerful tool to identify the root causes for stack overflow, memory leak or even data abort scenarios.

Last but not least, the Intel JTAG Debugger provides OS awareness, the ability to debug dynamically loaded services  and in-depth chipset register visibility via bitfield editors. Dedicated windows display all currently active kernel threads including their status and a view of all currently loaded kernel modules. The combination of these strengths is tailored towards being a strong support for device driver testing and development.

The advanced breakpoint dialog is easily accessible from the Intel JTAG Debugger GUI. It allows you to set conditional breakpoints, link actions like variable assignments to hitting a breakpoint. Furthermore it allows you to only have the execution flow halted then a breakpoint is hit for nth time. It even allows you to specify the use of hardware breakpoints for memory areas (like flash memory) where software breakpoints cannot be set. All the features available for code breakpoints are also available for data breakpoints or watchpoints with the additional choice of stopping execution on read or write or both types of data/variable access. Breakpoints can of course also be set by simply double-clicking on a source line or by entering a breakpoint (hard or soft, data or code) in the command window.

It is also good to have the ability to go directly into the memory window and modify variables and/or RAM memory contents on the fly to influence execution flow. The memory window is freely modifiable. To simplify understanding of the data structures it represents, it permits view of the memory contents in a variety of number formats, so the memory contents can be grouped in chunks that represent the data you expect there. To apply small code fixes on the fly to test them and validate them, you are able to change the opcode or modify the assembly code in the disassembly window. This is a feature that you obviously want to use with some care, but it can be very powerful to test a small fix before doing a complete rebuild.

Once you loaded the symbol info using for example the command

LOAD /SEGMENT /DEBUG /NOLOAD /GLOBAL OF "/usr/linux-2.6.34.i686/vmlinux"

or using the load dialog from the debugger GUI, the debugger proceeds to read the vmlinux .elf file on the host and uses the debug records and symbol maps from that file. This makes it important to ensure that the vmlinux image loaded into the debugger is identical to the one used to create the compressed image file for your Linux* OS that is actually being unpacked and running on the target device. In other words, we face a similar scenario as mentioned earlier for firmware and bootloader debug. If your build environment host and your debug host differ the symbol info in the vmlinux file may point to the wrong directory for your sources. To ensure that all the sources for the OS kernel are found correctly, you can use the set directories dialog that you can find form the debugger options pull down menu.

You can also use the debugger command set dir /subst="source location as stored in the vmlinux file" "source location on debug host", to achieve the same thing. If you opt not to do a directory substitution in this scenario, you may be asked to locate a source file the debugger finds symbol info for several times during the debug process until the debugger created all the substitution rules required on it's own. Thus it is more convenient to do the directory substitution on connect up front.

Since the Intel AtomProcessor supports hyperthreading, the OS kernel you are developing most likely has SMP enabled and is running two system hyper-threads. The Intel JTAG Debugger connects to both hyperthreads unless you tell it not to by modifying the startup script xdb.sh. If you are debugging a non-SMP enabled Linux* core, the entry for SMP support in the debugger connection command string inside of xdb.sh needs to be removed, since SMP support is the default setting. The debugger's behavior in SMP enabled mode is very similar in terms of execution control and breakpoints  to that expected on a single threaded system. There are a few details worth being aware of though:

The debugger takes care of all aspects regarding the fact that we work on a system with multiple hw-threads available. E.g. it will set hw-breakpoints and handle sw-breakpoint hits on all hw-threads.
When the target is stopped, the debugger shows the context of the hyper-thread which hit the last debug exception.
Run, Stop, Step and Next commands will control both cores. (The only exception here is the single instruction step which was explicitly called by the user; this will only step one instruction on one hyper-thread)
Whenever the context is switched to another hyper-thread (debug exception on a hyper-thread different than the current one), the user is informed by a console print "Context switch to SMP execution unit X" (where X is the number of the hyper-thread).

The on-chip instruction trace feature the Intel JTAG Debugger provides in addition to all this can be very useful to track down the root cause for data aborts or segmentation faults triggered by some OS component under development.  You can set a watchpoint on the OS signal sig variable and tell the debugger to halt on a specific OS signal. As soon as the signal for a segmentation fault is being issued by the Linux OS kernel (sig=11), the debugger will stop execution and you can then use the execution trace to analyze the code leading up to this potentially fatal exception.

Device Drivers and Kernel Modules:

This issue of tracking down segmentation faults leads us to yet another and higher level step in the task of debugging the system level software stack on an Intel Atom processor-based platform. One of the most likely culprits in causing segmentation faults tends to be kernel modules and device drivers under development. Developing and debugging kernel modules and device drivers are one of the key steps towards a a complete and stable OS platform. Easy debuggability of these device drivers is very desirable, though usually hard to find.

More than likely during the device driver adaptation and development phase you will work with kernel modules to shorten the rebuild and deploy cycle. Being able to debug dynamically runtime loaded kernel modules without having to instrument any code, and thus possibly alter its execution behavior, is thus very desirable. Taking advantage of the Intel Debugger's OS awareness, this can be done by using only two key steps that are required to enable kernel module debugging.

1) Have the kernel module idbntf.ko that is provided with the system debugger available on the target platform. Load the kernel module. It will handle passing information about kernel module load and unload activities as well as information abut the kernel module memory load location to the debugger, thus enabling runtime loaded kernel module debugging.

2) Set the directories where symbol info can be found for both the debugger as a whole and the Linux* OS awareness plugin.

In addition, for developing, analyzing and debugging device driver and OS service code, it is very important and useful to understand the interactions of the device driver code with the hardware device implementation on the platform chipset. This gets us back to the integrated bitfield editor.

The Intel Debugger assists with this by providing a detailed bitfield view of each and every single one of the 400 or more peripheral registers on the Intel Atom processor and chipset.

Through a treeview-based dialog grouped by device and hardware feature classes, you can select a specific register and open the bitfield editor for this register. The bitfield editor gives you detailed bitwise visibiltiy. You can see for each and every bit and monitor how the bit values change with the execution of the device driver you would like to debug. The bitfield editor also comes with a detailed description of the purpose of a register and what the settings of different bits mean.

Known Issues or Limitations:

Please be aware that although the Intel Application Debugger and Intel JTAG Debugger have been tested against Linux* kernels including version 2.6.34 and against some MeeGo* preview releases, they have currently not undergone full in-depth validation against any public MeeGo* releases.

Please consult the release notes for a detailed known current limitations listing of our debug solution.

Summary: 

With the Intel Application Debugger and Intel JTAG Debugger for Intel Atom processor included in the Intel Embedded Software Development Tool Suite 2.2 for Intel Atom processor Intel offers a comprehensive debug solution for Linux* running on a wide variety of devices from netbooks and tablet slates, to Connect TV, handhelds and embedded solutions. Their rich feature set and simple GUI-driven approach to debugging on all levels of the software stack enables a unified debug experience and reduced time and effort to identifying and resolving those issues on an Intel Atom processor-based platform that may delay software release.