Dear all,
I am trying to run the current MRCC binary (2020-02-22) with hybrid MPI/OpenMP parallelism using the SLURM batch queueing system.
Interactively (without batch queueing), MRCC runs fine as expected. E.g., with "mpitasks=2" and "OMP_NUM_THREADS=40" I obtain 2 MPI tasks via hydra_pmi_proxy, each spawning 40 threads, running successfully to completion using scf_mpi and mrcc_mpi.
With SLURM batch queueing, however, this appears to conflict with SLURM's "srun" command: dmrcc seems to launch instances of "srun" which are calling Intel MPI (hydra_pmi_proxy), and then hangs at the scf_mpi process.
Has anyone else encountered this issue, and perhaps come up with a solution?
Cheers,
Martin
Below is the slurm script, which runs fine interactively via
Code:
> salloc --overcommit --nodes=1 --ntasks=2 --cpus-per-task=40
salloc: Node node45-008 ready for job
> ssh node45-008
> ./mrcc.sh
but hangs when submitted via "sbatch mrcc.sh":
mrcc.sh:
Code:
#!/bin/bash
#SBATCH --overcommit
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=40
export OMP_NUM_THREADS=40
export MKL_NUM_THREADS=40
MRCC_DIR=/compuchem/bin/mrcc/2020-02-22.binary
INTEL_DIR=/cluster/intel/2019.5/compilers_and_libraries_2019.5.281/linux
source ${INTEL_DIR}/mkl/bin/mklvars.sh intel64 ilp64
source ${INTEL_DIR}/mpi/intel64/bin/mpivars.sh release_mt
export PATH=${MRCC_DIR}:${PATH}
cp ${HOME}/mrcc.inp MINP
dmrcc &> ${HOME}/mrcc.out
exit
The stalled/hanging processes on the scheduled node:
Code:
> ssh node45-001
> ps xl
PID PPID STAT TTY TIME COMMAND
132202 132197 S ? 0:00 /bin/bash /var/spool/slurm/d/job480438/slurm_script
132225 132202 S ? 0:00 /bin/sh /cluster/intel/2019.5/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin/mpirun -np 1 dmrcc_mpi
132231 132225 S ? 0:00 mpiexec.hydra -np 1 dmrcc_mpi
132232 132231 Ssl ? 0:00 /usr/bin/srun -N 1 -n 1 --nodelist node45-008 --input none /cluster/intel/2019.5/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin//hydra_bstrap_proxy --upstream-host node45-008.cm.cluster --upstream-port 45452 --pgid 0 --launcher slurm --launcher-number 1 --base-path /cluster/intel/2019.5/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin/ --tree-width 16 --tree-level 1 --time-left -1 --collective-launch 1 /cluster/intel/2019.5/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
132233 132232 S ? 0:00 /usr/bin/srun -N 1 -n 1 --nodelist node45-008 --input none /cluster/intel/2019.5/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin//hydra_bstrap_proxy --upstream-host node45-008.cm.cluster --upstream-port 45452 --pgid 0 --launcher slurm --launcher-number 1 --base-path /cluster/intel/2019.5/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin/ --tree-width 16 --tree-level 1 --time-left -1 --collective-launch 1 /cluster/intel/2019.5/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
132246 132240 S ? 0:00 /cluster/intel/2019.5/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
132249 132246 S ? 0:00 dmrcc_mpi
132516 132231 Ss ? 0:00 /usr/bin/srun -N 1 -n 1 --nodelist node45-008 --input none /cluster/intel/2019.5/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin//hydra_bstrap_proxy --upstream-host node45-008.cm.cluster --upstream-port 46735 --pgid 1 --launcher slurm --launcher-number 1 --base-path /cluster/intel/2019.5/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin/ --tree-width 16 --tree-level 1 --time-left -1 --collective-launch 1 /cluster/intel/2019.5/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9
