Help with Installing NGSolve on Supercomputer with Limited Permissions

Thauwa · September 22, 2019, 5:31am

Hello,

I am attempting to install your great software on a CentOS supercomputer. However, I have limited root/sudo/yum permissions, like most normal users on the framework. I would have to go through a big level of administrative stuff to get them to give me necessary permissions, or for them to install it for me, because I am just a PhD student.

Currently, I am lost trying to install NGSolve. I tried the following over several days:

install per instructions on: https://ngsolve.org/docu/latest/install/installlinux.html: failed due to lack of permissions needed to install dependencies. CMake fails.
tried to package it with all dependencies into a container using Singularity or Docker: failed because my machine is 32-bit and runs on Windows (I tried VirtualBox for a Virtual Linux VM, but still could not run Singularity). Docker won’t even install because it needs a 64-bit machine.
tried to find luck with Spack: Package List — Spack 0.20.0.dev0 documentation : failed because it seemed to install only Netgen but not NGSolve. Is there a way for NGSolve to be put up there too? Or is it already there and I am missing something?

I would really appreciate any help/advice with installing NGSolve on the computer system. I am moving onto the next stage of my research project and need supercomputing power.
Thanks very much for your time.

Thauwa · September 22, 2019, 10:04pm

An update:

I saw the following forum posts:

and changed my CMake code to:

cmake \ -DUSE_GUI=OFF \ -DUSE_PYTHON=ON \ -DUSE_MUMPS=ON \ -DUSE_PARDISO=ON \ -DUSE_UMFPACK=ON \ -DUSE_MPI=ON \ -DMKL_ROOT=${MKLROOT} \ -DUSE_MKL=ON \ -DCMAKE_INSTALL_PREFIX=${BASEDIR}/ngsolve-install ${BASEDIR}/ngsolve-src

Prior to all this, I load:

module load gcc/8.2.1 module load cmake/3.11.2 module load icc_18 module load contrib/python/3.6.3 module load anaconda3_5.3

but cannot install other dependencies due to lack of permissions.

I also used a pre-contained Docker image of NGSolve (Docker) using Singularity, but really could not figure out how to set the path in such a way to call my Python code in a way like: netgen testngsolve.py.

Any help would be appreciated.

A sample error message is:

[code]/opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_pair.h(218): note: this candidate was rejected because mismatch in count of arguments
_GLIBCXX_CONSTEXPR pair()
^
detected during:
instantiation of “std::pair<_T1, _T2>::pair(std::piecewise_construct_t, std::tuple<_Args1…>, std::tuple<_Args2…>) [with _T1=const std::string, _T2=ngcore::detail::ClassArchiveInfo, _Args1=<const std::string &>, _Args2=<>]” at line 136 of “/opt/rh/devtoolset-8/root/usr/include/c++/8/ext/new_allocator.h”
instantiation of “void __gnu_cxx::new_allocator<_Tp>::construct(_Up *, _Args &&…) [with _Tp=std::_Rb_tree_node<std::pair<const std::string, ngcore::detail::ClassArchiveInfo>>, _Up=std::pair<const std::string, ngcore::detail::ClassArchiveInfo>, _Args=<const std::piecewise_construct_t &, std::tuple<const std::string &>, std::tuple<>>]” at line 475 of “/opt/rh/devtoolset-8/root/usr/include/c++/8/bits/alloc_traits.h”
instantiation of “void std::allocator_traits<std::allocator<_Tp>>::construct(std::allocator_traits<std::allocator<_Tp>>::allocator_type &, _Up *, _Args &&…) [with _Tp=std::_Rb_tree_node<std::pair<const std::string, ngcore::detail::ClassArchiveInfo>>, _Up=std::pair<const std::string, ngcore::detail::ClassArchiveInfo>, _Args=<const std::piecewise_construct_t &, std::tuple<const std::string &>, std::tuple<>>]” at line 639 of
“/opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_tree.h”
instantiation of “void std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_M_construct_node(std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_Link_type, _Args &&…) [with _Key=std::string, _Val=std::pair<const std::string, ngcore::detail::ClassArchiveInfo>, _KeyOfValue=std::_Select1st<std::pair<const std::string, ngcore::detail::ClassArchiveInfo>>, _Compare=std::lessstd::string, _Alloc=std::allocator<std::pair<const std::string, ngcore::detail::ClassArchiveInfo>>,
_Args=<const std::piecewise_construct_t &, std::tuple<const std::string &>, std::tuple<>>]” at line 654 of “/opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_tree.h”
instantiation of “std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_Link_type std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_M_create_node(_Args &&…) [with _Key=std::string, _Val=std::pair<const std::string, ngcore::detail::ClassArchiveInfo>, _KeyOfValue=std::_Select1st<std::pair<const std::string, ngcore::detail::ClassArchiveInfo>>, _Compare=std::lessstd::string, _Alloc=std::allocator<std::pair<const std::string, ngcore::detail::ClassArchiveInfo>>,
_Args=<const std::piecewise_construct_t &, std::tuple<const std::string &>, std::tuple<>>]” at line 2414 of “/opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_tree.h”
instantiation of “std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::iterator std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_M_emplace_hint_unique(std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::const_iterator, _Args &&…) [with _Key=std::string, _Val=std::pair<const std::string, ngcore::detail::ClassArchiveInfo>, _KeyOfValue=std::_Select1st<std::pair<const std::string, ngcore::detail::ClassArchiveInfo>>, _Compare=std::lessstd::string,
_Alloc=std::allocator<std::pair<const std::string, ngcore::detail::ClassArchiveInfo>>, _Args=<const std::piecewise_construct_t &, std::tuple<const std::string &>, std::tuple<>>]” at line 501 of “/opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_map.h”
instantiation of “std::map<_Key, _Tp, _Compare, _Alloc>::mapped_type &std::map<_Key, _Tp, _Compare, _Alloc>::operator[](const std::map<_Key, _Tp, _Compare, _Alloc>::key_type &) [with _Key=std::string, _Tp=ngcore::detail::ClassArchiveInfo, _Compare=std::lessstd::string, _Alloc=std::allocator<std::pair<const std::string, ngcore::detail::ClassArchiveInfo>>]” at line 28 of “/usr/lusers/tharindu/ngsuite/ngsolve-src/external_dependencies/netgen/libsrc/core/archive.cpp”

compilation aborted for /usr/lusers/tharindu/ngsuite/ngsolve-src/external_dependencies/netgen/libsrc/core/archive.cpp (code 2)
make[8]: *** [libsrc/core/CMakeFiles/ngcore.dir/build.make:63: libsrc/core/CMakeFiles/ngcore.dir/archive.cpp.o] Error 2
make[7]: *** [CMakeFiles/Makefile2:1097: libsrc/core/CMakeFiles/ngcore.dir/all] Error 2
make[6]: *** [Makefile:163: all] Error 2
make[5]: *** [CMakeFiles/netgen.dir/build.make:116: dependencies/src/netgen-stamp/netgen-build] Error 2
make[4]: *** [CMakeFiles/Makefile2:132: CMakeFiles/netgen.dir/all] Error 2
make[3]: *** [Makefile:130: all] Error 2
make[2]: *** [CMakeFiles/netgen_project.dir/build.make:111: dependencies/Stamp/netgen_project/netgen_project-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:142: CMakeFiles/netgen_project.dir/all] Error 2
make: *** [Makefile:152: all] Error 2
[/code]

lkogler · September 30, 2019, 1:10pm

On a supercomputer, I think compiling NGSolve yourself is probably the only feasible option.

I am unsure if anaconda python works with MPI-parallel NGSolve.
-DUSE_GUI=OFF turns off the gui, so “netgen” will not be built. I presume the “netgen” in your case is left over from an old install.
-DUSE_GUI=ON and -DUSE_MPI=ON will compile netgen, but the GUI will still not work with MPI.

3.5) With -DUSE_GUI=OFF, some prerequisites from the documentation are not necessary anymore (tcl/tk, libglu, …)

When you are missing prerequisites and have no root permissions, I am afraid you have to compile them yourselfs (or ask your admin if they can install whatever you need for you).
When -DUSE_MPI=ON, we are installing a script called “ngspy” to the “bin” directory in the install directory. You can use that instead of “python3” if you run into issues with the loading of libraries. You can have a look at the script, it just preloads a couple of libraries before executing python3.

I hope that helps. If you run into issues with compiling, post them here, maybe I can help.

Best,
Lukas

Thauwa · September 30, 2019, 4:07pm

Hello Lukas,

Thank you for your reply. I have two follow-up questions:

If I build the installation with DUSE_GUI=OFF, should I change the code in my Python files? I ask because you mentioned netgen won’t be built. In particular, how would the following change?

import netgen.gui
from ngsolve import *
from netgen.csg import *
import numpy as np
geo = CSGeometry()

Also, based on Parallel computing with NGS-Py — NGS-Py 6.2.1803 documentation, will I have issues generating meshes (and with other basic functionality), if I use MPI? In other words, will my code still work if I use MPI and still contained:

cube = OrthoBrick( Pnt(-1,-1,-1), Pnt(1,1,1) ).bc("outer") geo.Add (cube) mesh = Mesh(geo.GenerateMesh(maxh=mesh_size))?

If I install NGSolve without the GUI, how do I run python files with it? Without asking me first, the admins made a Singularity image for me (using default website instructions), which opens the GUI on the supercomputer. However, if I run “netgen testfile.py”, it says netgen is not found, as expected. Therefore I tried loading the python file directly via the GUI menu, but then the GUI crashed (saying it ran out of memory; but we tested memory use and turns out it really did not run out of memory). Regardless, let’s say the admins built a Singularity image for me with an NGSolve installation using:

cmake \
-DUSE_GUI=OFF \ 
-DUSE_MUMPS=ON \
-DUSE_PARDISO=ON \
-DUSE_UMFPACK=ON \
-DUSE_MPI=ON \
-DMKL_ROOT=${MKLROOT} \
-DUSE_MKL=ON \
-DCMAKE_INSTALL_PREFIX=${BASEDIR}/ngsolve-install ${BASEDIR}/ngsolve-src

How do I call run a Python file with this installation?
Will using this installation change the answer to 1)?

Thank you very much for your time!

lkogler · September 30, 2019, 4:31pm

There is no netgen executable with DUSE_GUI=off. Instead of “netgen script.py” run “python3 script.py” or “ngspy script.py”. In python scripts, just remove import netgen.gui. At the top of your python scripts it should say “from ngsolve import *” or something similar.

Meshing is in general not MPI parallel. But, you can generate a mesh on rank 0 and then distribute it, or alternatively load it from a file. Consult the example files in ngsolve-src-dir/py_tutorials/mpi/ or the MPI-tutorials from this year’s usermeeting (link under news->3rd ngsolve usermeeting).

Besides that, adaptive refinement and multigrid do not work with MPI.

When it says that netgen is not found, but you have it installed, the reason is usually that you have not added ngsolve-install-dir/bin to the PATH environment variable.