install NGsolve without root access

Main Forum Old-Forum - do not post here
21

The variable MKL_THREADING_LAYER was set to GNU
I turned on or off MKL_MULTI_THREADED, the error still appears.
I run make test_ngsolve, only 3 out of 23 passed. they are assemble.py, taskmanager.py, cpuusage

Here is the complete list in CmakeCache.txt related to MKL:

//Path to a library.
MKL_BLACS_LIBRARY:FILEPATH=/soft/intel/x86_64/2015/composer_xe_2015_msi/composer_xe_2015.3.187/mkl/lib/intel64/libmkl_blacs_lp64.a

//Path to a library.
MKL_CORE_LIBRARY:FILEPATH=/soft/intel/x86_64/2015/composer_xe_2015_msi/composer_xe_2015.3.187/mkl/lib/intel64/libmkl_core.so

//Path to a library.
MKL_FFT_LIBRARY:FILEPATH=/soft/intel/x86_64/2015/composer_xe_2015_msi/composer_xe_2015.3.187/mkl/lib/intel64/libmkl_cdft_core.so

//Path to a file.
MKL_INCLUDE_DIR:PATH=/soft/intel/x86_64/2015/composer_xe_2015_msi/composer_xe_2015.3.187/mkl/include

//Path to a library.
MKL_INTERFACE_LIBRARY:FILEPATH=/soft/intel/x86_64/2015/composer_xe_2015_msi/composer_xe_2015.3.187/mkl/lib/intel64/libmkl_intel_lp64.so

//Use threaded MKL libs
MKL_MULTI_THREADED:BOOL=ON

//No help, variable specified on the command line.
MKL_ROOT:UNINITIALIZED=/soft/intel/x86_64/2015/composer_xe_2015_msi/composer_xe_2015.3.187/mkl

//Path to a library.
MKL_SCALAPACK_LIBRARY:FILEPATH=/soft/intel/x86_64/2015/composer_xe_2015_msi/composer_xe_2015.3.187/mkl/lib/intel64/libmkl_scalapack_lp64.so

//Link single dynamic MKL lib
MKL_SDL:BOOL=OFF

//Link static MKL
MKL_STATIC:BOOL=OFF

// Threading layer
MKL_THREADING_LAYER:UNINITIALIZED=GNU

//Path to a library.
MKL_THREADING_LIBRARY:FILEPATH=/soft/intel/x86_64/2015/composer_xe_2015_msi/composer_xe_2015.3.187/mkl/lib/intel64/libmkl_gnu_thread.so

22

Sorry for being unclear, I meant to set the environment variable MKL_THREADING_LAYER to GNU, i.e. in your terminal:
export MKL_THREADING_LAYER=GNU

23

I did that also… no change in the error.

24

Hmm, this is something we have to fix…

In the meantime you could try two other things:
cmake -DMKL_MULTI_THREADED=OFF … at configuration
(acutally, if you use 1 MPI rank per CPU core, that makes sense)

or:
have a look at the ngspy script generated. There are some libraries assigned to LD_PRELOAD before calling python. You could add the path go libgomp.so there (for me it’s /usr/lib/libgomp.so)

Regards,
Matthias

25

Hi,

I also had some troubles with MKL on our cluster. For me using statically linked libraries worked wonders…

have a try with -DMKL_STATIC=ON

Best,
Christoph

26

I added -DMKL_MULTI_THREADED=OFF, and finally I have a working ngsolve installed! Big thanks to both of you, Lukas and Matthias.

Here is what’s going on:
I have a build directory, and a do-configure file that include all the CMake options.
To do a new compile, I just remove the old files in CMakeCache.txt and CMakeFiles/,
and run ./do-configure (is this not the correct way to configure ??!!)
Anyway, this seems to be wrong and some old files related to my MKL setup was not modified appropriately.
After creating a new folder and compile from scratch, the installation was successful.

Best,
Guosheng

27

I’m glad you got it finally running. Thinking about it, MKL_STATIC=ON might have helped too, actually.
I know about the issue that you sometimes have to delete the whole build directory when you change the configuration. That’s also something we need to address.
I will go over the thread again and check how many of your issues we can work around automatically in the future. The remaining ones should at least be documented somewhere.

Thanks for your patience and have fun with NGSolve! :slight_smile:

Best,
Matthias

28

Hi Matthias,

Here is the summary of two main issues:

Problem 1: MPI not working properly.
Reason: mpich in my cluster is only compatible with gcc version 4.9.2, all other versions do not work.

MKL related issue
Problem 2.1: with dynamic library -DMKL_STATIC=OFF (default version)
(a) blacs library not found
(b) libmkl_gnu_thread.so: undefined symbol: omp_get_num_procs

Fix (a): Manually change the blacs library to libmkl_blacs_lp64.a in ccmake .
(b): turn off multi threading by adding flag -DMKL_MULTI_THREADED=OFF in cmake
In conclusion, with the command
-DMKL_ROOT=… -DMKL_MULTI_THREADED=OFF -DMKL_SDL=OFF
mkl is installed correctly and mumps and pardiso are available in ngsolve.

Problem 2.2: with static library -DMKL_STATIC=ON (Christoph’s suggestion)
This time the blacs library is found to be an openmpi version:
libmkl_blacs_openmpi_lp64.a
which is not compatible with my mpich, after manually changing it to libmkl_blacs_lp64.a in CMakeCache.txt
The code compiles with a multi threaded version of MKL.

Interestingly, I ran the same tests
mpirun -n 5 ngspy mpi_cmagnet.py, and observe a much faster speed with the dynamic MKL library than the static library. Don’t understand why, though…

There is a final problem with building ngsolve with hyper, I got some error message, but I just gave up with hyper since pardiso and mumps are good enough for me right now :wink:

Here is the complete list of my build (python3 is installed in ~/local.bin):

export SRC_DIR=~/netgen/src
export INSTALL_DIR=~/local
export GCC_DIR=/panfs/roc/msisoft/gcc/4.9.2_2
export LD_LIBRARY_PATH=$GCC_DIR/lib/:$GCC_DIR/lib64:$GCC_DIR/lib/gcc/x86_64-unknown-linux-gnu/4.9.2:$LD_LIBRARY_PATH
export PATH=$INSTALL_DIR/bin:$GCC_DIR/bin/:$PATH

module load mpich
module load cmake
export LD_LIBRARY_PATH=$INSTALL_DIR/lib:$LD_LIBRARY_PATH

cmake
-DCMAKE_C_COMPILER=$GCC_DIR/bin/gcc
-DCMAKE_CXX_COMPILER=$GCC_DIR/bin/g++
-DCMAKE_PREFIX_PATH=$INSTALL_DIR
-DCMAKE_BUILD_TYPE=Release
-DINSTALL_DIR=$INSTALL_DIR
-DUSE_GUI=OFF
-DUSE_MPI=ON
-DUSE_MUMPS=ON
-DUSE_HYPRE=OFF
-DUSE_MKL=ON
-DMKL_ROOT=…/mkl
-DMKL_SDL=OFF
-DMKL_MULTI_THREADED=OFF
$SRC_DIR/ngsolve

Best,
Guosheng