Running with MPI and OpenMP

Hello,
I have recently compiled MRCC with OpenMP and MPI, however, the MPI jobs always faily with the error "forrtl: severe (174): SIGSEGV, segmentation fault occurred". I invoked build.mrcc with "Intel -pOMP -pMPI=OpenMPI -f/project/6000595/cdc/Software/MRCC_2020" using Intel/2020.1.217, OpenMPI/4.0.3, and Libfabric/1.10.1. All other test jobs completed fine, only the MPI ones failed with the same error - leading me to believe that I am missing an environmental variable or something like that. I have currently set the following variables:
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export MKL_NUM_THREADS=$SLURM_CPUS_PER_TASK
export OMP_SET_MAX_ACTIVE_LEVELS=$SLURM_CPUS_PER_TASK
export OMP_PLACES=CORES
export OMP_PROC_BIND=SPREAD,CLOSE
export OMPI_MCA_RMAPS_BASE_INHERIT='1'
export OMPI_MCA_HWLOC_BASE_BINDING_POLICY='CORE'
export OMPI_MCA_BTL_OPENIB_ALLOW_IB='TRUE'

If more information is needed, please let me know. I have spent a long time trying to figure this out, and at this point, I think I'm just over my head.

Thanks a bunch,
Colton

Dear Colton,

I am sorry to hear about your problems.

The first idea is about your Open MPI version. Please, note that
some patches might be needed for OpenMPI (see point 8 on page 29 of the manual).

It would probably be simpler to try IntelMPI instead of OpenMPI, to my knowledge the
runtime Intel MPI Library can be downloaded for free.

Did you try the provided precompiled binary MRCC package?
That also supports MPI runs. Does the binary exhibit the same behaviour?

Could you please also look at Sect. 9.3 of the manual and the following forum threads?
www.mrcc.hu/index.php/forum/running-mrcc...penmp-and-slurm#1031
www.mrcc.hu/index.php/forum/compiling-mr...u-openmpi-pcm-xc#925

If none of that helps I would definitely need more information.
You should share at least a complete output file and corresponding error message.
If jobs fail at a different stage, then please provide one example of each type.
The above forum threads list a number of other parameters that we usually need to help.

Which feature/method of MRCC are you interested in? I am asking because that module
may not even support MPI parallelism.

I hope some of it helps,
Peter

Hello,

I wish to perform CCSD and CCSD(T) optimizations and frequency calculations.

I emailed support regarding whether the OpenMPI has the fixed commits or not and they do not know - so I will assume that it does not.

I tried the binary with avx512 and IntelMPI (the 2019.3 version), but kept getting a segmentation fault and a single-line output file for the MPI calculations:
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx512/Compiler/intel2019/intelmpi/2019.3.199/intel64/bin/mpirun: line 104: 241283 Segmentation fault (core dumped) mpiexec.hydra "$@" 0<&0

I am in a bit of a time-crunch to submit these calculations, so I am just wondering if I would be better off just sticking to using only OpenMP. (As you can probably tell, I don't know a whole lot about this stuff, and we are an experimental group so I don't have anyone in my group to ask).

Thanks a bunch for your help,

Colton

Hello,

The support for our cluster was able to fix the OpenMPI problems and the program now runs smoothly.

Thanks for your help though!

Colton

Dear Colton,

great, I am glad it worked out.
Could you, please, share in more detail what seemed to be the problem and the solution so that the community can learn from this? Was that the missing patches?

Thank you and I hope our program will be useful for you.
Best wishes,
Peter

I will ask support and relay what he says - I am not entirely sure exactly what he all did.