Booting a third party OS on MIC

Booting a third party OS on MIC

Hi there:

Recently I'm trying to port Barrelfsih, which is a multi-kernel operating system designed for many-core platform by MS and ETH, to Xeon Phi. I start with booting a simple program at uos_loadoffset(0x4000000) in the aperture space of KNC2250 device to test if a third-party program could run on Xeon Phi. The programs is wirtten with C as:

int a;
int b;
int main()
    a = 4;
    b = 5;
    goto loop;
    return 0;

then it's compiled: gcc -o main -m32 -fno-builtin -nostdlib -e main main.c

the linker script "" is:

. = ALIGN(4k);
.text 0x4000000 : ALIGN(4k)

. = ALIGN(4k);
.rodata . :

.bss . :

/***** These sections get discarded *****/
/* Discard exception handler frames and headers -- we don't use em */
/* *(;*/
/* *(.dynsym); */
/* *(.dynstr); */
/* *(.hash); */
/* *(.gnu.hash); */

after the compiling, the elf-format executable file "main" is "objcopy"ed: objcopy -O binary -S -R -R .comment main main.img

so, at last the binary image "main.img" is as:

Disassembly of section .text:

04000000 <main>:
4000000: 55                            push %ebp
4000001: 89 e5                        mov %esp,%ebp
4000003: c7 05 04 10 00 04 04 movl $0x4,0x4001004
400000a: 00 00 00
400000d: c7 05 00 10 00 04 05 movl $0x5,0x4001000
4000014: 00 00 00
4000017: eb fe                         jmp 4000017 <main+0x17>

Disassembly of section .bss:

04001000 <b>:
4001000: 00 00 add %al,(%eax)

04001004 <a>:
4001004: 00 00 add %al,(%eax)

here, the variables of a(@0x4001004) and b(@0x4001000) will be set to be 4 and 5 respectively when the binray image get executed on Xeon Phi.

After  "main.img" get produced, I download it to the uos_loadoffset(0x4000000) in the aperture space of KNC2250 and tell Xeon Phi to execute the binary code thought function boot_linux_uos(...).The I try to read the values at offset 0x4001004 and 0x4001000 in the aperture space of KNC2250 to check if they are set to be 4 and 5 respectively, which means that the binary code is executed by Xeon Phi, but it doesn't live up to my expectation:(, all the values I got are just 0x00000000, so don't know if the binary code got executed. What's more, after a while, the Xeon Phi card(5110P) got so hot that I couldn't evern touch it.

So, Any body get some experiences of writting code running directly on Xeon Phi without OS supporting? Any body could tell me if the binray code at 0x4000000 in the aperture space of KNC2250 get executed? Is there some other facilities on board such as LEDs, buzzer or Serial Port that I could use to monitor if the binary code is exectuing!

Thanks in advance.


10 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Hello Yaung,

that is an interesting behaviour. From the documentation I though it should be easy to boot something on the XeonPhi. Did you have more success meanwhile? 

Why do you use objcopy to create a binary? Is that a way to create something that looks like a bzImage for the fboot1 bootloader? Maybe its possible to throw an ELF image on it instead, because the Intels maintainance kernel is just an ELF image...


Thanks for reply. Do you mean that the ELF image(@0x4000000) can be directly executed by the Xeon phi? So I need not to "objcopy" the ELF image to be a bzImage style image? But one more question, how can I check if the program is running properly? I cann't read the aperture space to check the value in the memory of the 5110P card when programs start to run on it!

When experimenting with your own O/S, and if you are using the passive cooled model of Xeon Phi, try to set you fan speed (cooling the Xeon Phi) to high, at least until you can get the system management code to boost your fan speeds when necessary.

Jim Dempsey

I had some more success over the weekend. The fboot1 bootloader handles 64bit ELF images perfectly well. You set up your image with the physical addresses where you want to have your code, for example at the 1Mb position. The 0x4000000 address is only for communication with the bootloader and you do not need to fiddle with that directly. Just cat "boot:elf:/root/boot64.elf" > /sys/class/mic/mic0/state and the driver will do its magic. Just remember that you have to provide 32bit code for the entry point although it is a 64bit ELF. Access to the aperture space works only after you used this boot mechanism, not before!

The host-side driver will wait for a response from your OS and otherwise times out. In order to signal a successful boot, you have to set bit 1 of the SBOX_SCRATCH2 register. But to write there, you first have to set up a page table to the MMIO address range...

And please believe Jim's comment. The XeonPhi does not much on its own to power down when it gets hot. Normally, you would get a CPUHOT interrupt and then reconfigure fans, frequency, and voltage manually. But the early OS code has probably all interrupts disabled.

Thank you so much! I'll follow your advice and tell you the result as soon as I get it! Currently, there is too little technical materials I can reference to for system programming on Xeon-phi, so your help is greatly appreciated!

Thank Jim for the advice, but there is no fan on my card. So maybe I can just set the processor to run at a lower frequency when it gets too hot.

The Xeon Phi cards do have a separate monitoring processor (8,16, 32-bit?) that communicates over (a separate channel on) the PCIe bus. The motherboard, also has a separate monitoring processor. This little network (SMBus) is supposed to have code that includes fan control. Generally the code that runs on the coprocessor reports temperatures across (out) the SMBus and the host monitoring processor captures the readings and adjusts the fans. In many cases, monitoring processor on the motherboard does nothing with the data other than making it available to the hosts main processor. And then on the host system you have a driver that periodically reads the temperatures and tweaks the fan speeds. Until you can address the code in the motherboards monitoring processor and/or the driver/app on the host side, manually set your fan speeds (to the Xeon Phi) to high.

Jim Dempsey

Hello Yaung,

some news from my experiments: After loading an 64bit ELF image, the bootloader stays in 64bit mode. Thus you can ignore the 32bit entry point and directly start with 64bit code :)


Hi Randolf, Yaung!

I'm working on the same thing for PathScale as part of a new API and I'm interested in any working code you could share. While I came up with a somewhat working solution, I'd like to cross-check my work and maybe learn some things about setting up page tables or share my knowledge about reducing ELF memory footprint of the binary.

By the way - thanks for confirming that the boot loader uses 64-bit entry point, this runs contrary to the documentation.

I'm also interested in rebuilding the uOS bzimage in MPSS 3.1 - does anyone have any pointers? Some DMA initialization is done on the device side and since it's not clear from the docs, I need to rebuild the kernel until I find it.



Another thread that may help is



Leave a Comment

Please sign in to add a comment. Not a member? Join today