proper environment setup for MPI in jupyter notebook

Main Forum Old-Forum - do not post here
1

Hello,

How to properly set up jupyter notebook so that I can use MPI there? I couldn’t find a solution online.
For python, we simply preload the library using the ngspy hack:

LD_PRELOAD=... pythonr $*

Can we do the same thing for jupyter notebook?

Best,
Guosheng

2

Are you really sure this is something you absolutely want to do?

As far as the preloads are concerned, If you set the LD_PRELOAD from ngspy as an environment variable it should work. But that will not yet allow you to run MPI parallel jupyter notebooks - the jupyter notebook spawns a python instance in the background, and you want THAT to be parallel, not the notebook itself.

I have managed to set something like this up in the past using jupyter clusters, but is is very finicky and a bit unstable.

You can look into jupyter clusters and have a look at the MPI jupyter notebooks from the 2nd user meeting if you want to give it a try. I can also give you the ipyparallel config files I used back then if you want.

Best,
Lukas

3

Hi Lukas,

Looks very complicated to do… ;<
I am doing jupyter only for education purpose. I just want to show the petsc amg preconditioner from the ngs-petsc library, which needs MPI to start with.
Currently, I simply go back to the plain python code to do the demo.
Can you walk me thought the setup? :>

Best,
Guosheng

4

The petsc-amg preconditioner needs an NGSolve version that is compiled with MPI, but it does not require you to actually run in parallel.

If you want to use this in a sequential run (but an MPI version), just setting the LD_LIBRARY_PATH should do it. It only gets complicated when you want multiple MPI ranks.

If this does not work for some reason, a workaround would be to manually load the libraries from the ngspy script at the top of your jupyter notebook. Have a look at CDLL from the ctypes python module.

Best,
Lukas

5

I just want to use the preconditioner in serial.
Setting LD_LIBRARY_PATH in my bashrc file does not work.
Can you give me a demo on manual load via ngspy?
CDLL and ctypes in python are foreign language to me. I am afraid dig into that.

6

Put something like this at the top of the jupyter notebook

from ctypes import CDLL, RTLD_GLOBAL
# these are the libraries that are preloaded by ngspy, we can replace all the MKL libraries by
# the runtime library libmkl_rt
libs = ["/home/lkogler/local/openmpi-3.1-gcc-8.1/lib/libmpi.so", "/opt/intel/mkl/lib/intel64/libmkl_rt.so"]
for lib in libs:
    CDLL(lib, RTLD_GLOBAL)

from ngsolve import *

Generally, I have found that you really only need to preload the libmkl_rt library when you do it this way.

Best,
Lukas

7

Hi Lukas,

Thanks for the magic lines, it’s all working now.
In my case, I have to preload libmpi.so and libpetsc.so. My ngspy doesn’t preload libmkl_rt.so. (NO idea why this is the case, but the code runs anyway)

Also, I have a side question related to jupyter notebook: I would like to print out the convergence history and iteration counts for CGSolver, but it didn’t show up in the notebook. Instead, the iteration counts are printed out in my terminal. Any suggestion how to bring these output to notebook?

Best,
Guosheng

8

In ngsolve.krylovspace there is a CGSolver python class. If you use that you should be able to get that information.
Best

9

Ah-ha, there are two CGSolvers… didn’t notice that, lol
Thanks, Christopher.

10

Hello Christopher,

I would like to ask, how to install NGSolve MPI/PETESC in Anaconda (win10)?

Thanks in advance!

Libo Yan

11

Hi, sorry there is no windows mpi version yet. Afaik petsc there is no petsc package for win in conda as well. If you want to try NGSolve with petsc I suggest a virtual machine
Best

12

Hello Ikogler,

must we include “intel MPI” in the proper environment setup for MPI in jupyter notebook?