跳转至

gcc mpi 源码编译

gcc mpi 源码编译过程记录

GCC

下载源码

Configure

注意：

  • 不能在 src 目录 configure，而是应该另外创建一个 build 目录
  • gcc 提供了依赖安装脚本，不用手动去下载
cd /tmp/dir
wget https://mirrors.ustc.edu.cn/gnu/gcc/gcc-8.5.0/gcc-8.5.0.tar.xz
tar xf gcc-8.5.0.tar.xz
cd gcc-8.5.0
#安装依赖
./contrib/download_prerequisites

#在build目录下configure
cd ..
mkdir build
cd build

../gcc-8.5.0/configure --disable-multilib --prefix=$HOME/ENV/gcc-8.5.0 --enable-languages=c,c++

make -j 2>&1 |tee make.log #32线程编译大概40分钟左右
make install
  • --disable-multilib：gcc 默认支持编译 32 位程序 (-m32)，关闭此功能

解决链接库错误

原因，程序是动态链接的。运行时使用的是系统的库。而系统的库和编译时使用的库不兼容。

  • 方法一：修改LD_LIBRARY_PATH环境变量

    export LD_LIBRARY_PATH=$HOME/opt/gcc-9.1.0/lib64

  • 方法二：编译时将库搜索路径存在二进制文件中

    ~/opt/gcc-9.1.0/bin/g++ -Wl,-rpath=$HOME/opt/gcc-9.1.0/lib64 --std=c++17 demo.cc -o demo

  • 方法三：静态链接

    ~/opt/gcc-9.1.0/bin/g++ --std=c++17 demo.cc -o demo -static-libstdc++ -static-libgcc

遇到的问题

解压源码很慢

服务器上解压 60M 的源码非常慢，解压了 10-20 分钟。不只是解压，cp, rm 源码目录都非常的慢。推测是 NFS 的原因。

解决方法为使用本地目录，如/tmp

gcc8.3.0 make 失败

[Building gcc 8.3 Makefile:955: all] Error 2 - Stack Overflow

好像是 8.3.0 自身的问题，换用 8.5.0 编译成功

GCC 使用

GLIBC_xxx Not Found

查看 glibc 版本

  • 20.04: GLIBC_2.30
  • 22.04: GLIBC_2.35 
    strings /lib/x86_64-linux-gnu/libc.so.6 |grep GLIBC

几乎所有软件都会动态连接到 glibc 库，因此软件如果是在高版本的机器上编译的，那么在低版本机器上运行就会报错。

strings /staff/fyyuan/install/mpich/lib/libmpi.so.12.4.2 |grep GLIBC |tail
__isoc99_fscanf@GLIBC_2.7
toupper@GLIBC_2.2.5
lseek@GLIBC_2.2.5
ferror@GLIBC_2.2.5
getifaddrs@GLIBC_2.3
clock_getres@GLIBC_2.17
openat@GLIBC_2.4
qsort@GLIBC_2.2.5
lstat@GLIBC_2.33
fwrite@GLIBC_2.2.5

MPI 使用

遇到的问题

mpi4py 依赖固定路径的 libmpi.so

已经确定 PATH、LIB_LIBRARY_PATH 中没有自己编译的路径了。运行一个简单的测试仍然固定依赖于我的 libmpi.so

(base) ➜  ~ mpiexec -n 2 python test_mpi2.py
Traceback (most recent call last):
  File "test_mpi2.py", line 1, in <module>
    from mpi4py import MPI
ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /staff/fyyuan/install/mpich/lib/libmpi.so.12)
Traceback (most recent call last):
  File "test_mpi2.py", line 1, in <module>
    from mpi4py import MPI
ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /staff/fyyuan/install/mpich/lib/libmpi.so.12)

当前 python 为 conda 的 base，其中 mpi4py 是用 pip 安装的。发现 conda 重新安装即可 

 pip uninstall mpi4py
Found existing installation: mpi4py 4.0.0
Uninstalling mpi4py-4.0.0:
  Would remove:
    /staff/fyyuan/miniconda3/lib/python3.8/site-packages/mpi4py-4.0.0.dist-info/*
    /staff/fyyuan/miniconda3/lib/python3.8/site-packages/mpi4py/*
Proceed (Y/n)? y
  Successfully uninstalled mpi4py-4.0.0

conda install mpi4py

神奇的是它把 mpiexec 更改了 

which mpiexec
/staff/fyyuan/miniconda3/bin/mpiexec

自己编译的 

~/install/mpich-4.2.2/bin/mpiexec --version
HYDRA build details:
    Version:                                 4.2.2
    Release Date:                            Wed Jul  3 09:16:22 AM CDT 2024
    CC:                              gcc
    Configure options:                       '--disable-option-checking' '--prefix=/staff/fyyuan/install/mpich-4.2.2' '--with-hwloc=embedded' 'CC=gcc' 'CXX=g++' '--disable-fortran' '--enable-fast=all,O3' '--cache-file=/dev/null' '--srcdir=../../../../mpich-4.2.2/src/pm/hydra' 'CFLAGS= -DNDEBUG -DNVALGRIND -O3' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -DNETMOD_INLINE=__netmod_inline_ucx__ -I/staff/fyyuan/build/mpich-4.2.2-build/src/mpl/include -I/staff/fyyuan/build/mpich-4.2.2/src/mpl/include -I/staff/fyyuan/build/mpich-4.2.2/modules/json-c -I/staff/fyyuan/build/mpich-4.2.2-build/modules/json-c -I/staff/fyyuan/build/mpich-4.2.2/modules/hwloc/include -I/staff/fyyuan/build/mpich-4.2.2-build/modules/hwloc/include -D_REENTRANT -I/staff/fyyuan/build/mpich-4.2.2-build/src/mpi/romio/include -I/staff/fyyuan/build/mpich-4.2.2/src/pmi/include -I/staff/fyyuan/build/mpich-4.2.2-build/src/pmi/include -I/staff/fyyuan/build/mpich-4.2.2-build/modules/yaksa/src/frontend/include -I/staff/fyyuan/build/mpich-4.2.2/modules/yaksa/src/frontend/include'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Demux engines available:                 poll select

它的 

mpiexec --version
HYDRA build details:
    Version:                                 3.3.2
    Release Date:                            Tue Nov 12 21:23:16 CST 2019
    CC:                              x86_64-conda_cos6-linux-gnu-cc -I/staff/fyyuan/miniconda3/include -I/staff/fyyuan/miniconda3/include -L/staff/fyyuan/miniconda3/lib -Wl,-rpath,/staff/fyyuan/miniconda3/lib
    CXX:                             x86_64-conda_cos6-linux-gnu-c++ -I/staff/fyyuan/miniconda3/include -I/staff/fyyuan/miniconda3/include -L/staff/fyyuan/miniconda3/lib -Wl,-rpath,/staff/fyyuan/miniconda3/lib
    F77:                             x86_64-conda_cos6-linux-gnu-gfortran -I/staff/fyyuan/miniconda3/include -L/staff/fyyuan/miniconda3/lib -Wl,-rpath,/staff/fyyuan/miniconda3/lib
    F90:                             x86_64-conda_cos6-linux-gnu-gfortran -I/staff/fyyuan/miniconda3/include -L/staff/fyyuan/miniconda3/lib -Wl,-rpath,/staff/fyyuan/miniconda3/lib
    Configure options:                       '--disable-option-checking' '--prefix=/staff/fyyuan/miniconda3' '--disable-dependency-tracking' '--enable-cxx' '--enable-fortran' '--disable-wrapper-rpath' 'MPICHLIB_CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/staff/fyyuan/miniconda3/include -fdebug-prefix-map=/tmp/build/80754af9/mpich-mpi_1575396103289/work=/usr/local/src/conda/mpich-3.3.2 -fdebug-prefix-map=/staff/fyyuan/miniconda3=/usr/local/src/conda-prefix' 'MPICHLIB_CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/staff/fyyuan/miniconda3/include' 'MPICHLIB_CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/staff/fyyuan/miniconda3/include -fdebug-prefix-map=/tmp/build/80754af9/mpich-mpi_1575396103289/work=/usr/local/src/conda/mpich-3.3.2 -fdebug-prefix-map=/staff/fyyuan/miniconda3=/usr/local/src/conda-prefix' 'MPICHLIB_FFLAGS=-fopenmp -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/staff/fyyuan/miniconda3/include -fdebug-prefix-map=/tmp/build/80754af9/mpich-mpi_1575396103289/work=/usr/local/src/conda/mpich-3.3.2 -fdebug-prefix-map=/staff/fyyuan/miniconda3=/usr/local/src/conda-prefix' 'MPICHLIB_FCFLAGS=-fopenmp -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/staff/fyyuan/miniconda3/include -fdebug-prefix-map=/tmp/build/80754af9/mpich-mpi_1575396103289/work=/usr/local/src/conda/mpich-3.3.2 -fdebug-prefix-map=/staff/fyyuan/miniconda3=/usr/local/src/conda-prefix' 'CC=x86_64-conda_cos6-linux-gnu-cc' 'CFLAGS=-I/staff/fyyuan/miniconda3/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/staff/fyyuan/miniconda3/include -fdebug-prefix-map=/tmp/build/80754af9/mpich-mpi_1575396103289/work=/usr/local/src/conda/mpich-3.3.2 -fdebug-prefix-map=/staff/fyyuan/miniconda3=/usr/local/src/conda-prefix -O2' 'LDFLAGS=-L/staff/fyyuan/miniconda3/lib -Wl,-rpath,/staff/fyyuan/miniconda3/lib' 'CPPFLAGS=-I/staff/fyyuan/miniconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/staff/fyyuan/miniconda3/include -I/tmp/build/80754af9/mpich-mpi_1575396103289/work/src/mpl/include -I/tmp/build/80754af9/mpich-mpi_1575396103289/work/src/mpl/include -I/tmp/build/80754af9/mpich-mpi_1575396103289/work/src/openpa/src -I/tmp/build/80754af9/mpich-mpi_1575396103289/work/src/openpa/src -D_REENTRANT -I/tmp/build/80754af9/mpich-mpi_1575396103289/work/src/mpi/romio/include' 'CPP=/staff/fyyuan/miniconda3/bin/x86_64-conda_cos6-linux-gnu-cpp' 'CXX=x86_64-conda_cos6-linux-gnu-c++' 'CXXFLAGS=-I/staff/fyyuan/miniconda3/include -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/staff/fyyuan/miniconda3/include -fdebug-prefix-map=/tmp/build/80754af9/mpich-mpi_1575396103289/work=/usr/local/src/conda/mpich-3.3.2 -fdebug-prefix-map=/staff/fyyuan/miniconda3=/usr/local/src/conda-prefix -O2' 'FC=x86_64-conda_cos6-linux-gnu-gfortran' 'FCFLAGS=-I/staff/fyyuan/miniconda3/include -fopenmp -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/staff/fyyuan/miniconda3/include -fdebug-prefix-map=/tmp/build/80754af9/mpich-mpi_1575396103289/work=/usr/local/src/conda/mpich-3.3.2 -fdebug-prefix-map=/staff/fyyuan/miniconda3=/usr/local/src/conda-prefix -O2' 'FFLAGS=-I/staff/fyyuan/miniconda3/include -fopenmp -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/staff/fyyuan/miniconda3/include -fdebug-prefix-map=/tmp/build/80754af9/mpich-mpi_1575396103289/work=/usr/local/src/conda/mpich-3.3.2 -fdebug-prefix-map=/staff/fyyuan/miniconda3=/usr/local/src/conda-prefix -O2' '--cache-file=/dev/null' '--srcdir=.' 'LIBS=' 'MPLLIBNAME=mpl'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Checkpointing libraries available:
    Demux engines available:                 poll select

MPI

mpi 运行时需要保证版本一致。而 apt 安装的 mpich 受发行版版本影响，可能不一样。因此最好的方式是自己从源码编译 mpich，放在 NFS 等公共可以访问的地方。

MPI 编译较快（<半小时），可以编译很多版本使用

mpich 和 openmpi 对比

MPICH：Guides | MPICH

  • 支持 mpi2.0，功能强大，效率高
  • 缺点是仅支持以太网

OpenMPI：

  • 支持 mpi2.0
  • 对于 CPU 核心较多的节点，推荐使用 openmpi
  • 支持各种网络，包括以太网和 infiniband

mpich 编译

p.s 切记要在低 GLIBC 版本的机器上编译，因为编译出来的结果都会动态链接到当前平台的 glibc 库（/lib/x86_64/libc.so），如果平台版本高（比如 ubuntu22.04 是 GLIBC 2.2.35），在低版本平台就无法运行 mpi 程序

官网有详细文档

  • 从源码编译：MPICH Installers’ Guide is a guide to help with the installation process of MPICH. Both Unix and Windows installation procedures are outlines.
  • MPICH Users’ Guide provides instructions to use MPICH. This manual explains how to run MPI applications after MPICH is installed and working correctly.

编译命令

  • 需要单独的 build 目录，和 src 目录分离
cd /path/to/build_tmp/

/path/to/src/configure -prefix=/path/to/install 2>&1 | tee c.txt

make -j 2>&1 | tee m.txt   # debug 可以 VERBOSE=1

make install 2>&1 | tee mi.txt

配置后的例子

../mpich-4.2.2/configure --prefix=$HOME/install/mpich-4.2.2 CC=gcc CXX=g++ --disable-fortran --enable-static --enable-fast=all,O3 2>&1 |tee tee c.log

configure 选项

  • --disable-fortran
    • 文档里写的是--disable-f77：禁用 fortran 77，--disable-fc：禁用 Fortran 90（及之后 95, 2003, 2008 版本）但是尝试后无效，仍然会要求你有 fortran 编译器。
  • --enable-fast MPICH libraries are built with default compiler optimization, -O2, which can be modified by --enable-fast configure option. For instance, --disable-fast disables the default optimization option. --enable-fast=O sets default compiler optimization as -O<n>
  • --enable-static：是否产生 libmpi.a 等文件
  • --enable-shared
    • mpicc 编译出来的程序使用动态链接，运行时需要设置LD_LIBRARY_PATH。
    • --disable-shared --enable-static 则不用设置LD_LIBRARY_PATH

enable-fast

Both xFLAGS and MPICHLIB xFLAGS affect the compilation of the MPICH libraries. However, only xFLAGS is appended to MPI wrapper scripts, mpicc and friends.

MPICH libraries are built with default compiler optimization, -O2, which can be modified by --enable-fast configure option. --enable-fast=O<n> sets default compiler optimization as -On

./configure --enable-fast=all,O3

./configure --enable-fast=all MPICHLIB_CFLAGS=-O3 \ 
            MPICHLIB_FFLAGS=-O3 \ 
            MPICHLIB_CXXFLAGS=-O3 \ 
            MPICHLIB_FCFLAGS=-O3

第二种：This will cause the MPICH libraries to be built with -O3, and -O3 will not be included in the mpicc and other MPI wrapper script

Process Manger

  • --with-pm=hydra

MPICH has been designed to work with multiple process managers; that is, although you can start MPICH jobs with mpiexec, there are different mechanisms by which your processes are started. An interface (called PMI) isolates the MPICH library code from the process manager. Currently three process managers are distributed with MPICH

  • hydra: This is the default process manager that natively uses the existing daemons on the system such as ssh, slurm, pbs.
  • gforker: This is a simple process manager that creates all processes on a single machine. It is useful both for debugging and for running on shared memory multiprocessors.

–with-pm Select the process manager. The default is hydra; also useful are gforker and remshell. You can build with all three process managers by specifying --with-pm=hydra:gforker:remshell

mpiexec 选项

mpiexec -n 5 -f machinefile /path/to/build_tmp/examples/cpi
mpiexec -n 5 -hosts icarus3,icarus4 /path/to/build_tmp/examples/cpi

Usage: ./mpiexec [global opts] [local opts for exec1] [exec1] [exec1 args] : [local opts for exec2] [exec2] [exec2 args] : ...
  • 可以一次运行多个 exec，参数使用 : 分隔
  • global opt 对所有 exec 生效
    • -f {name}: file containing the host names
    • -hosts {host list}: comma separated host list
    • Hydra specific options
  • local opt 对单个 exec 生效
    • -n/-np {value}: number of processes
    • {exec_name} {args}: executable name and arguments

遇到的问题

cluster 上执行时使用了错误的 hydra_pmi_proxy

自己编译的 mpich 在 cluster 上执行时报错。

mpirun -f ~/app/machinefile.i3 -n 64 ~/build/mpich-build/examples/cpi

zsh:1: no such file or directory: /usr/bin/hydra_pmi_proxy

-verbose 可以展示 Hydra 的详细信息（mpiexec --help 查看更多）

可以看到最后执行的命令，由于 /usr/bin/hydra_pmi_proxy 不存在，因此就报错了。

[mpiexec@icarus4] Launch arguments: /usr/bin/ssh -x icarus3 "/usr/bin/hydra_pmi_proxy" --control-port icarus4:45175 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id 0
zsh:1: no such file or directory: /usr/bin/hydra_pmi_proxy

调用/usr/bin/hydra_pmi_proxy 而不是自己编译产生的 hydra_pmi_proxy 的·原因是，没有正确把 PATH 环境变量传递给 remote。

但是我已经在 zshrc 中设置了 PATH 使用编译的 mpich。发现原来是当前的 shell 没有 source bashrc/zshrc 导致的。source 后问题解决。

UCX version

configure: error: UCX installation does not meet minimum version requirement (v1.9.0). Please upgrade your installation, or use --with-ucx=embedded

hydra

make 时报错 

make[4]: Entering directory '/staff/fyyuan/build/mpich-4.2.2-build/src/pm/hydra'
  CCLD     mpiexec.hydra
/usr/bin/ld: /staff/fyyuan/build/mpich-4.2.2-build/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `hwloc_get_next_obj_by_type':
/staff/fyyuan/build/mpich-4.2.2/modules/hwloc/include/hwloc/inlines.h:98: undefined reference to `hwloc_get_type_depth'

对应的 configure 

../mpich-4.2.2/configure --prefix=$HOME/install/mpich-4.2.2 CC=gcc CXX=g++ --disable-fortran --enable-fast=all,O3 2>&1 |tee tee c.log

openmpi

../openmpi-4.1.4/configure --prefix=$HOME/ENV/openmpi-4.1.4 --disable-mpi-fortran CC=gcc CXX=g++

运行选项

-v verbose
-q quiet
--use-hwthread-cpus  #使用物理核作为slot数目

-H, -host, --host <host1,host2,...,hostN>
-hostfile, --hostfile <hostfile>
    % cat myhostfile
    aa slots=2
    bb slots=2
    cc slots=2

-c, -n, --n, -np <#>
-cpus-per-proc, --cpus-per-proc <#perproc>
-bind-to-core, --bind-to-core
-bind-to-socket, --bind-to-socket

遇到的问题

run
➜  workspace mpirun -n 4 ./mpi_hello
--------------------------------------------------------------------------
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default.  The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.

  Local host:              snode0
  Local adapter:           mlx5_0
  Local port:              1

--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   snode0
  Local device: mlx5_0
--------------------------------------------------------------------------
Hello world from processor snode0, rank 3 out of 4 processors
Hello world from processor snode0, rank 0 out of 4 processors
Hello world from processor snode0, rank 1 out of 4 processors
Hello world from processor snode0, rank 2 out of 4 processors
[snode0:2773927] 3 more processes have sent help message help-mpi-btl-openib.txt / ib port not selected
[snode0:2773927] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[snode0:2773927] 3 more processes have sent help message help-mpi-btl-openib.txt / error in device init