Are shared libraries with offload code "fat" libraries?

Are shared libraries with offload code "fat" libraries?

When compiling a C++ program with a section of code that is "#pragma offload target(mic) {...some code...}", it appears that the compiled code for "some code" is automatically transfered to the co-processor and run there. This is what I find in the documentation, and it works as expected. Apparently, the resulting binary contains the code for both host and co-processor, being a "fat binary" http://en.wikipedia.org/wiki/Fat_binary

When I compile a shared library that contains code declared "#pragma offload_attribute(push, target(mic))", my experiments seem to indicate that the resulting shared library contains both host and co-processor code, and that it is tranferered to the coprocessor automatically when called. Apparently the .so file is also a "fat" binary. However, I cannot find any documentation on this. The documentation seems to suggest to always build pure (thin) MIC libraries when a certain library is required on the coprocessor (using -mmic).

Questions:

  • Is the observation about fat shared libraries correct (or is the compiler/linker doing some magic such as inlining certain code)?
  • I seem not to find any documentation on this behaviour. Where is it documented?
  • Is there some way to convice the compiler to consider all code to be within a "#pragma offload_attribute(push, target(mic))" offload section, or do I really have to add this to all my files?
  • Is there any way to observe in detail which shared libraries are required/attempted to be loaded on the coprocessor?

Thanks,
Georg 

10 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

> Is there some way to convice the compiler to consider all code to be within a "#pragma offload_attribute(push, target(mic))" offload section, or do I really have to add this to all my files?

You can use the "-offload-attribute-target=MIC" on the compile line to compile a whole file for both host and MIC.

http://software.intel.com/sites/products/documentation/doclib/stdxe/2013... 

Quote:

James Cownie (Intel) wrote:

You can use the "-offload-attribute-target=MIC" on the compile line to compile a whole file for both host and MIC.

Just what I needed to answer question 4. Thank you! Georg

Ronald W Green (Intel)'s picture

> Is the observation about fat shared libraries correct (or is the compiler/linker doing some magic such as inlining certain code)?

Yes, it's a fat binary

> I seem not to find any documentation on this behaviour. Where is it documented?

compiler documentation for Phi specifics: http://software.intel.com/en-us/articles/programming-and-compiling-for-intel-many-integrated-core-architecture

Thanks! However, I did not see any fat shared library information in the reference you posted, nor in the other documentation that I saw.

Georg

I have been trying this with libz. I compiled a version of libz using the recipe shown below, basically compiling everything with -offload-attribute-target=mic. When I run an executable that is using this libz.so, I get "The remote process indicated that the following libraries could not be loaded: libz.so.1.2MIC.5. offload error: cannot load library to the device 0 (error code 19)". Adding the location of this library to MIC_LD_LIBRARY_PATH plus creating a copy of the library via " cp libz.so.1.2.5 libz.so.1.2MIC.5" changes the error message to "offload error: cannot load library to the device 0 (error code 22)".

Questions:

  1. How can I check that my libz.so.1.2.5 as generated by the procedure below is indeed a "fat" library? (maybe something went wrong)
  2. What is the exact meaning of error 22?
  3. Why does it request a library named  "libz.so.1.2MIC.5" instead of using the "fat" shared library that it generated?

Thanks for the help.

Georg

Recipe for compiling libz to be a "fat" library:

$ CC=icc CFLAGS="-offload-attribute-target=mic -fPIC" LDSHARED="icc -shared" ./configure --prefix=`pwd`/compiled_offload --64
$ make
# Creation of fat lib actually failed, because mechanism in makefile creates .o files in objcts/ and moves
# only the .o files are copied to .lo, not the *MIC.o files. do link step manually:
$ icc -shared -offload-attribute-target=mic -fPIC -D_LARGEFILE64_SOURCE=1 -o libz.so.1.2.5 adler32.o compress.o crc32.o deflate.o gzclose.o gzlib.o gzread.o gzwrite.o infback.o inffast.o inflate.o inftrees.o trees.o uncompr.o zutil.o -lc
$ make test
$ make install

And as an additional question:

5. How can I observe what shared libraries the host side runtime system is trying to transfer to the MIC? I am asking because a small test program seems to work properly with my "fat" libz. But I find it difficult to understand what is going wrong with my full executable.

Georg

robert-reed (Intel)'s picture

I don't know much about the inner workings of "fat" libraries, but I see something else in your recipe that might be a problem.  The line above that specifies offload builds for the compiler is used in the invocation of ./configure, which works by test compiling and running simple codes to determine parameters of the SW architecture.  Configure is expected to run on the local architecture but doesn't know anything about "offload."  If configure runs in the scenario described above, it will collect host-side statistics, not those of the target. 

I have seen other threads that describe the process of building native libraries for the Intel MIC architecture, but all I can find at the moment is this post:, http://software.intel.com/en-us/forums/topic/359610, which describes a novel way to use host-side cross compilers to mimic native compilation (this otherwise requires setting up a native compilation environment on the coprocessor in order to accommodate inquisitors like ./configure).  I haven't tried this method so I can't vouch for it, but I think native library compilation approaches will get you to where you want to go faster for libraries that rely on ./configure to set their environment.

Ravi Narayanaswamy (Intel)'s picture

You can use the tool mic_extract  on your fat binary.

mic_extract  fat.out

This will extract the MIC binary from fat.out  into fatMIC.out

The hint with mic_extract was really useful. It allowed me to verify that my libz.so in fact is a fat binary, and that my exectuable has references to libz.so.1.2MIC.5 on its MIC side.

In the end, it turned out that the problem was a wrong LD_LIBRARY_PATH setting in the environment (not MIC_LD_LIBRARY_PATH !) that caused the executable to pick up a libz.so that contained no MIC side code. Apparently,  the runtime system falls back to search for a libz.so.1.2MIC.5 if the necessary MIC side code cannot be extracted from the shared library.

I think it would really be helpful if there were tools/documentation that would help to understand what is going on in the case of fat offload binaries. This problem cost me 2.5 days, and I think that's too much for such a trivial issue:

  1. documentation. It should be documented somewhere how the runtime system handles fat libraries/executables (using LD_LIBRARY_PATH, falling back to pure MIC lib, ...), how to check if executables/shared libs are actually "fat", what they contain. The equivalents of "ldd" and "nm" should be documented. Document  error messages of the runtime system and possible causes. Also the "inner workings" of fat library handling should be explained to a certain extent.
  2. add tools: the equivalent for "nm" and "ldd" for fat binaries/libs.
  3. add debugging facilities: Some way to monitor the code load process onto MIC, so we can determine at which point it goes wrong.

Georg 

Login to leave a comment.