× If you have problems during the execution of MRCC, please attach the output with an adequate description of your case as well as the followings:
  • the way mrcc was invoked
  • the way build.mrcc was invoked
  • the output of build.mrcc
  • compiler version (for example: ifort -V, gfortran -v)
  • blas/lapack versions
  • as well as gcc and glibc versions

This information really helps us during troubleshooting :)

Running with MPI and OpenMP

More
3 weeks 3 days ago #1053 by cdc
Hello,
I have recently compiled MRCC with OpenMP and MPI, however, the MPI jobs always faily with the error "forrtl: severe (174): SIGSEGV, segmentation fault occurred". I invoked build.mrcc with "Intel -pOMP -pMPI=OpenMPI -f/project/6000595/cdc/Software/MRCC_2020" using Intel/2020.1.217, OpenMPI/4.0.3, and Libfabric/1.10.1. All other test jobs completed fine, only the MPI ones failed with the same error - leading me to believe that I am missing an environmental variable or something like that. I have currently set the following variables:
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export MKL_NUM_THREADS=$SLURM_CPUS_PER_TASK
export OMP_SET_MAX_ACTIVE_LEVELS=$SLURM_CPUS_PER_TASK
export OMP_PLACES=CORES
export OMP_PROC_BIND=SPREAD,CLOSE
export OMPI_MCA_RMAPS_BASE_INHERIT='1'
export OMPI_MCA_HWLOC_BASE_BINDING_POLICY='CORE'
export OMPI_MCA_BTL_OPENIB_ALLOW_IB='TRUE'

If more information is needed, please let me know. I have spent a long time trying to figure this out, and at this point, I think I'm just over my head.

Thanks a bunch,
Colton

Please Log in or Create an account to join the conversation.

More
3 weeks 2 days ago #1054 by nagypeter
Dear Colton,

I am sorry to hear about your problems.

The first idea is about your Open MPI version. Please, note that
some patches might be needed for OpenMPI (see point 8 on page 29 of the manual).

It would probably be simpler to try IntelMPI instead of OpenMPI, to my knowledge the
runtime Intel MPI Library can be downloaded for free.

Did you try the provided precompiled binary MRCC package?
That also supports MPI runs. Does the binary exhibit the same behaviour?

Could you please also look at Sect. 9.3 of the manual and the following forum threads?
www.mrcc.hu/index.php/forum/running-mrcc...penmp-and-slurm#1031
www.mrcc.hu/index.php/forum/compiling-mr...u-openmpi-pcm-xc#925

If none of that helps I would definitely need more information.
You should share at least a complete output file and corresponding error message.
If jobs fail at a different stage, then please provide one example of each type.
The above forum threads list a number of other parameters that we usually need to help.

Which feature/method of MRCC are you interested in? I am asking because that module
may not even support MPI parallelism.

I hope some of it helps,
Peter

Please Log in or Create an account to join the conversation.

More
3 weeks 1 day ago #1055 by cdc
Replied by cdc on topic Running with MPI and OpenMP
Hello,

I wish to perform CCSD and CCSD(T) optimizations and frequency calculations.

I emailed support regarding whether the OpenMPI has the fixed commits or not and they do not know - so I will assume that it does not.

I tried the binary with avx512 and IntelMPI (the 2019.3 version), but kept getting a segmentation fault and a single-line output file for the MPI calculations:
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx512/Compiler/intel2019/intelmpi/2019.3.199/intel64/bin/mpirun: line 104: 241283 Segmentation fault (core dumped) mpiexec.hydra "$@" 0<&0

I am in a bit of a time-crunch to submit these calculations, so I am just wondering if I would be better off just sticking to using only OpenMP. (As you can probably tell, I don't know a whole lot about this stuff, and we are an experimental group so I don't have anyone in my group to ask).

Thanks a bunch for your help,

Colton

Please Log in or Create an account to join the conversation.

More
3 weeks 1 day ago #1056 by cdc
Replied by cdc on topic Running with MPI and OpenMP
Hello,

The support for our cluster was able to fix the OpenMPI problems and the program now runs smoothly.

Thanks for your help though!

Colton

Please Log in or Create an account to join the conversation.

More
3 weeks 1 day ago #1057 by nagypeter
Dear Colton,

great, I am glad it worked out.
Could you, please, share in more detail what seemed to be the problem and the solution so that the community can learn from this? Was that the missing patches?

Thank you and I hope our program will be useful for you.
Best wishes,
Peter

Please Log in or Create an account to join the conversation.

More
3 weeks 5 hours ago #1058 by cdc
Replied by cdc on topic Running with MPI and OpenMP
I will ask support and relay what he says - I am not entirely sure exactly what he all did.

Please Log in or Create an account to join the conversation.

Time to create page: 0.023 seconds
Powered by Kunena Forum