Link problems with mkl under Linux

Link problems with mkl under Linux

Hi,

   I am facing a link problem with mkl under linux (Ubuntu 12.04) recently. I am trying to make my code work but it always tell me that there are link errors. The make file is:

%%%%%%%%%%%%%%% Makefile %%%%%%%%%%%%%

CC=g++ -DMKL_ILP64 -m64

MKLPATH = /opt/intel/composer_xe_2013_sp1.0.080/mkl

CFLAGS=-c -Wall

LIBS = -L$(MKLPATH)/lib/intel64

LDFLAGS= $(MKLPATH)/lib/intel64/libmkl_intel_ilp64.a -Wl,--start-group $(MKLPATH)/lib/intel64/libmkl_core.a  $(MKLPATH)/lib/intel64/libmkl_sequential.a  -Wl,--end-group  $(LIBS) -lpthread -lm

INCLUDES = -I$(MKLPATH)/include

SOURCES=main_gf.cpp  gfgraph.cpp  gfreader.cpp  gfsmw.cpp  gfcommunity.cpp  gfcomputation.cpp

OBJECTS=$(SOURCES:.cpp=.o)

EXECUTABLE=cdgreen

all: $(SOURCES)  $(EXECUTABLE)

$(EXECUTABLE): $(OBJECTS)

      $(CC) $(INCLUDES) $(OBJECTS) -W $(LIBS) $(LDFLAGS) -o $@

.cpp.o:

      $(CC) $(CFLAGS) $(INCLUDES) $< -o $@

clean:

       rm -rf *o $(EXECUTABLE)

%%%%%%%%%%%%%%% Makefile End %%%%%%%%%%%%%

I use the mkl link line advsor to get the link info I need, which is as follows:

“ use this link line:

-Wl,--start-group $(MKLROOT)/lib/intel64/libmkl_intel_ilp64.a $(MKLROOT)/lib/intel64/libmkl_core.a $(MKLROOT)/lib/intel64/libmkl_sequential.a -Wl,--end-group -lpthread –lm

Compiler options:

-DMKL_ILP64 -m64 -I$(MKLROOT)/include “

I have made these changes in my makefile as shown above.

In my code, in “gfcomputation.cpp” I use several mkl function, such as dgetrf_, dgetri_, cblas_dgemm etc.

When I run the make file, it  can generate object files well, but there are always link errors:

gfcomputation.o: In function `gfcomputation::matrixinv(double*, int)':

gfcomputation.cpp:(.text+0x1891): undefined reference to `dgetrf_(int*, int*, void const*, int*, int*, int*)'

gfcomputation.cpp:(.text+0x18b9): undefined reference to `dgetri_(int*, void const*, int*, int*, void const*, int*, int*)'

gfcomputation.o: In function `gfcomputation::matrixmulti(double*, double*, double*, int, int, int)':

gfcomputation.cpp:(.text+0x19b3): undefined reference to `cblas_dgemm(CBLAS_ORDER, CBLAS_TRANSPOSE, CBLAS_TRANSPOSE, int, int, int, double, double const*, int, double const*, int, double, double const*, int)'

gfcomputation.o: In function `gfcomputation::sparsematrixmulti(double*, double*, double*, int, int, int, int)':

gfcomputation.cpp:(.text+0x1c5b): undefined reference to `mkl_dcoomm(char*, int*, int*, int*, double*, char*, double*, int*, int*, int*, double*, int*, double*, double*, int*)'

collect2: ld returned 1 exit status

make: *** [cdgreen] Error 1

I searched on the web and it seems lots of people had this problems, but there’s no direct solutions to this problem. I also check the link example in mkl userguide, it seems my makefile is ok.

I have been struggling with this for several days and really need someone’ help. Thanks!

C.J.

 

 

16 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Hello C.J.

It looks that your code is using the 32 bit integer, but you are linking with 64-bit integer MKL libraries(ILP64 library). 
Note,for the 64 bit application, the default size for the integer is 32 bit.  So it needs to link with MKL LP64 libraries:

In the Makefile, can you try to remove '-DMKL_ILP64' from compiler flag, and change to link with 'libmkl_intel_lp64.a' library ( not libmkl_intel_Ilp64.a)?

LDFLAGS= $(MKLPATH)/lib/intel64/libmkl_intel_ilp64.a -Wl,--start-group $(MKLPATH)/lib/intel64/libmkl_core.a  $(MKLPATH)/lib/intel64/libmkl_sequential.a  -Wl,--end-group  $(LIBS) -lpthread -lm

Change to:  

LDFLAGS= -Wl,--start-group $(MKLPATH)/lib/intel64/libmkl_intel_lp64.a $(MKLPATH)/lib/intel64/libmkl_core.a  $(MKLPATH)/lib/intel64/libmkl_sequential.a  -Wl,--end-group  $(LIBS) -lpthread -lm

Thanks,
Chao

 

The first web search hit you should pay attention to would be the one on the MKL link advisor.  I suspect it will still tell you that all 3 libmkl_*.a need to be inside the start-group ..... end-group directives.  As far as libmkl_sequential is concerned, -lpthread should be unnecessary, and C++ should take care of the -lm even if it were required by a companion C compiler/linker.

Surround your prototype declarations for the MKL routines (and, more generally, routines intended to be called from C) with 

 extern "C" {
...prototype declaration...
}

Otherwise, the name decoration of C++ creates long external names (with argument types strung in) that do not exist in the MKL libraries.

You may reexamine the need for underscores appended to external names when building for 64-bit Linux.

Quote:

Chao Y (Intel) wrote:

Hello C.J.

It looks that your code is using the 32 bit integer, but you are linking with 64-bit integer MKL libraries(ILP64 library). 
Note,for the 64 bit application, the default size for the integer is 32 bit.  So it needs to link with MKL LP64 libraries:

In the Makefile, can you try to remove '-DMKL_ILP64' from compiler flag, and change to link with 'libmkl_intel_lp64.a' library ( not libmkl_intel_Ilp64.a)?

LDFLAGS= $(MKLPATH)/lib/intel64/libmkl_intel_ilp64.a -Wl,--start-group $(MKLPATH)/lib/intel64/libmkl_core.a  $(MKLPATH)/lib/intel64/libmkl_sequential.a  -Wl,--end-group  $(LIBS) -lpthread -lm

Change to:  

LDFLAGS= -Wl,--start-group $(MKLPATH)/lib/intel64/libmkl_intel_lp64.a $(MKLPATH)/lib/intel64/libmkl_core.a  $(MKLPATH)/lib/intel64/libmkl_sequential.a  -Wl,--end-group  $(LIBS) -lpthread -lm

Thanks,
Chao

 

Hi Chao,

     Thanks for your reply. I checked my machine using "uname -m" command and it tells me it's a 64-bit machine. But I still took your suggestion to change the LDFLAGS as you suggested and run the make file. Unfortunately it still didn't work. It shows the same error info. 

     After searching some material, I am not sure whether this is correct in my make file:

     "

      $(EXECUTABLE): $(OBJECTS)

               $(CC) $(INCLUDES) $(OBJECTS) -W $(LIBS) $(LDFLAGS) -o $@"

     I am not sure whether the position of $(OBJECTS) matters or not. I found a similar makefile here: http://blue.for.msu.edu/comp-notes/code/Makefile-example-1, which use :

    "

   example-1: example-1.o 
	     $(CXX) $(CXXFLAGS) $(OPTFLAGS) -o example-1 example-1.o  $(LIBS)

    It seems he put the .o file and $(LIBS) after the "-o". I am not very familiar with makefile so not sure whether I made a mistake here? Do you have any comments about this? THX!

C.J.

"

"

Quote:

mecej4 wrote:

Surround your prototype declarations for the MKL routines (and, more generally, routines intended to be called from C) with 

 

 extern "C" {
...prototype declaration...
}

 

Otherwise, the name decoration of C++ creates long external names (with argument types strung in) that do not exist in the MKL libraries.

You may reexamine the need for underscores appended to external names when building for 64-bit Linux.

Hi,

    Thanks for your reply. I changes the prototype declaration as follows:

extern "C" 

   void dlarnv(const int*, int*, const int*, double*); 
   void cblas_dgemm(const CBLAS_ORDER, const CBLAS_TRANSPOSE, const CBLAS_TRANSPOSE,  
                        const int, const int, const int, const double, const double *A, const int,  
                        const double *B, const int, const double, double*C, const int); 
   void dgetri (const int* n_mkl, double* Rin_mkl, const int* lda_mkl, const int* ipiv_mkl, double* work_mkl, const int* lwork_mkl, int* info_mkl); 
   void dgetrf (const int* m_mkl, const int* n_mkl, double* Rin_mkl, const int* lda_mkl, int* ipiv_mkl, int* info_mkl); 
   void mkl_dcoomm(char *TransA, int *M_mkl, int *N_mkl, int *K_mkl, double *alpha, char *matdescra, double *val, int *row,  
                       int *col, int *nnz, double *B, int *ldb, double *beta, double *c, int *ldc); 
}

     It should be ok now. But it still gave me the same link error: "undefined reference to...". Still trying to find where is the bug...

C.J.

I think that you should construct and post a small C source code that exhibits the problems that you encountered in your larger application. Without that available, we have to guess as to which MKL include files have been invoked and we cannot try to reproduce the problem on a similar machine/OS combination at this end.

If you want your ints to switch automatically from 32-bit for 32-bit mode build to 64-bit for 64-bit build on linux, you could define them as long int, or you could use the MKL headers and select a data type from them, which ought to help if you will be supporting more of the MKL target platforms.

I detect a hint that you expect plain int to become a 64-bit data type automatically.  That doesn't happen on any MKL target.  On the 64-bit mode targets, you have a choice of either lp64 (32-bit ints) or ilp64 (64-bit int) libraries, but it's your responsibility to choose consistent data types for your own code.

As stated in previous replies, we would need to see an actual sample of what you are doing.

Quote:

mecej4 wrote:

I think that you should construct and post a small C source code that exhibits the problems that you encountered in your larger application. Without that available, we have to guess as to which MKL include files have been invoked and we cannot try to reproduce the problem on a similar machine/OS combination at this end.

Hi,

    Thanks. I send you a message with a sample code cause I didn't find how to attache files in reply. Please have a look. Thanks very much!

C.J

Hi,

    Attached is my sample code and makefile. Please check it. Thanks.

C.J.

Attachments: 

AttachmentSize
Download testexample.zip1.3 KB

Change your linker rule in the makefile to

     $(CC) $(OBJECTS) $(LDFLAGS) $(LIBS) -o $@

so that the linker gets to note which symbols are needed by test.o before it searches the libraries. For more details, see the documentation for your system linker.

Is there a specific reason why you want to use static libraries?

Best Reply

C.J.

I had a few modifications for the makefile, it can work here:

LDFLAGS= -Wl,--start-group $(MKLPATH)/lib/intel64/libmkl_intel_lp64.a $(MKLPATH)/lib/intel64/libmkl_core.a  $(MKLPATH)/lib/intel64/libmkl_sequential.a  -Wl,--end-group -lpthread -lm –ldl

$(CC) $(LIBS) $(INCLUDES) $(OBJECTS) $(LDFLAGS) -o $@

Also,  in the source code, you can use:
#include <mkl_lapack.h>

to replace:
extern "C"
{ .......  }

Thanks,
Chao

Quote:

Chao Y (Intel) wrote:

C.J.

I had a few modifications for the makefile, it can work here:

LDFLAGS= -Wl,--start-group $(MKLPATH)/lib/intel64/libmkl_intel_lp64.a $(MKLPATH)/lib/intel64/libmkl_core.a  $(MKLPATH)/lib/intel64/libmkl_sequential.a  -Wl,--end-group -lpthread -lm –ldl

$(CC) $(LIBS) $(INCLUDES) $(OBJECTS) $(LDFLAGS) -o $@

Also,  in the source code, you can use:
#include <mkl_lapack.h>

to replace:
extern "C"
{ .......  }

Thanks,
Chao

Hi  Chao,

      Thanks so much for your help and my whole code can work now. It seems the main change is in the link line, you add "-Idl", which is not required from the link line advisor. What does this mean? 

C.J.

[quote=mecej4]

Change your linker rule in the makefile to

     $(CC) $(OBJECTS) $(LDFLAGS) $(LIBS) -o $@

so that the linker gets to note which symbols are needed by test.o before it searches the libraries. For more details, see the documentation for your system linker.

Is there a specific reason why you want to use static libraries?

Hi,

    Thanks for your help. Now through changing the makefile, the code can work now. Please refer to previous reply.

    I don't have special reason to use static libraries. Does it matters?

C.J.

Quote:

C.J.X. wrote:
 I don't have special reason to use static libraries. Does it matters?

Among others, three reasons to use static libraries are (i) slightly faster speed, (ii) no new bugs introduced by replaced shared libraries, and (iii) no shared libraries available, too much trouble to build them oneself. In early stages of program development, it is far better to use shared libraries as much as possible.

To sum up the content of this thread, the lesson is: for simple usage of MKL, (i) include MKL header files rather than creating function prototypes yourself, and (ii) use the -mkl compiler flag (/Qmkl in Windows).

The reason that your link required -ldl was that even though you linked to static MKL libraries, there were dependencies on other shared libraries (such as the g++ runtime) that required some symbols in the shared library support provided by libdl.so.

Quote:

mecej4 wrote:

Quote:

C.J.X. wrote: I don't have special reason to use static libraries. Does it matters?

Among others, three reasons to use static libraries are (i) slightly faster speed, (ii) no new bugs introduced by replaced shared libraries, and (iii) no shared libraries available, too much trouble to build them oneself. In early stages of program development, it is far better to use shared libraries as much as possible.

To sum up the content of this thread, the lesson is: for simple usage of MKL, (i) include MKL header files rather than creating function prototypes yourself, and (ii) use the -mkl compiler flag (/Qmkl in Windows).

The reason that your link required -ldl was that even though you linked to static MKL libraries, there were dependencies on other shared libraries (such as the g++ runtime) that required some symbols in the shared library support provided by libdl.so.

Hi,

    Thanks very much for your help!

C.J.

Leave a Comment

Please sign in to add a comment. Not a member? Join today