Restarting MR-CCSD

Dear Developers and Users of MRCC

I will run quite big computations at MR-CCSD level. However, before it, I test the restart functionality of MRCC code. I prepared two inputs (let call them A and B ). The B file contains "rest=1" command. The first job (A) started and worked mostly fine (I do not fully understand, why it did not stop after reaching ccmaxit iteration, but it is a rather minor problem). Anyway, It was kill due to the time limit. I copied the input B into the directory, which contains all files created during the execution of input A. Unfortunately, it seems that B does not use partly converged CC amplitudes from the file. Could you suggest the proper way of restarting MR-CCSD?

Below you can found the important parts of the outputs.

The output generated by job A:
**********************************************************************
MRCC program system
***************************************************************
(...)
Release date: February 22, 2020
Input file:
(...)
basis=cc-pVDZ
calc=CCSD
ccprog=mrcc
scftype=mcscf

occ=7,3,2,0/6,2,2,0
docc=5,1,1,0
mact=3,2,2,0
mem=15123MB
mult=3
charge=0
core=6
cctol=8
scftol=8
ccmaxit=1
maxmicroit=444
scfmaxit=222
verbosity=3

unit=bohr
geom
Al
F 1 R

R=3.1

Checking keyword combinations...

Warning! scfalg has been set to disk!
Keywords:

active=none
agrid=ld0006-ld0590
basis=cc-pvdz
basis_sm=none
basopt=off
bfbasis=none
bfgsmem=10
bfgstol=1.0d-3
bpcompo=0.985
bpcompv=0.98
bpdfo=0.985
bpocc=0.985
bppdo=0.985
bppdv=0.98
bpedo=0.985
bpedv=0.98
calc=ccsd
ccmaxit=1
ccprog=mrcc
ccsdalg=disk
ccsdmkl=seq
ccsdrest=off
ccsdthreads=2
cctol=8
charge=0
cialg=disk
ciguess=off
cmpgrp=auto
core=6
corembed=off
csapprox=off
dboc=off
dendec=cholesky
dens=0
dfalg=invsqrt
dfbasis_cor=none
dfbasis_scf=none
dfintran=ovirt
dft=off
dhexc=adc(2)
diag=david
docc=5,1,1,0
domrad=10.d0
drpaalg=fit
dual=off
ecp=auto
edisp=off
edisp_embed=off
embed=off
epert=none
eps=0.975
excrad=0.d0
excrad_fin=0.000000000D+00
fmm=off
fmmord=8
freq=off
gauss=spher
geom=zmat
gopt=off
ghost=none
gtol=7
grdens=off
grtol=10
hamilton=dc
iface=none
intalg=auto
itol=12
laptol=1.d-2
lccoporder=trffirst
lcorthr=normal
lccrest=off
lmp2dens=on
lnoepso=0.d0
lnoepsv=1e-6
localcc=off
localcorrsymm=off
locintrf=disk
mact=3,2,2,0
maxact=off
maxdim=100
maxex=0
maxmicroit=444
mcscfiguess=hf
mem=15123mb
molden=on
mpitasks=1
mulmet=0
mult=3
nacto=0
nactv=0
naf_cor=off
naf_scf=off
nafalg=albe
naftyp=
nchol=auto
ndeps=1e-3
nstate=1
nsing=0
ntrip=0
occ=7,3,2,0/6,2,2,0
oniom=off
oniom_eechg=off
oniom_pcm=off
optalg=
optmaxit=50
optetol=1e-6
optex= 0
optgtol=1e-4
optstol=1e-3
orblocc=off
orbloce=off
orbloco=off
orblocv=off
orblocguess=cholesky
osveps=1e-3
ovirt=off
ovltol=1e-7
ovosnorb=80.0
pcm=off
popul=off
pressure=100000
pssp=off
ptfreq=0.0
ptthreads=2
qscf=aughessg
qmmm=off
redcost_exc=off
redcost_tddft=off
refdet=none
rest=0
rgrid=log3
rohfcore=semicanonical
rohftype=standard
scfalg=disk
scfdamp=off
scfdiis=on
scfdiis_end=222
scfdiis_start=1
scfdiis_step=1
scfdtol=9
scfext=10
scfguessdens=
scfiguess=sad
scflshift=off
scfmaxit=222
scftype=mcscf
scftol=8
scspe=1.d0
scsph=1.d0
scsps=1.2d0
scspt=0.33333333333333
scspv=1.d0
spairtol=1e-4
subminp=top
symm=
talg=occ
temp=298.15
test=off
theodore=off
tlmo=0.999
tpao=0.94
tprint=off
uncontract=off
unit=bohr
usedisk=2
verbosity=3
wpairtol=0.100000000E-05
(...)
*** MCSCF ENERGY IN STEP 33 IS -341.3840944117161484 [AU]
======================================================================

SUCCESS...
THE SCF ITERATION HAS CONVERGED!

***FINAL MCSCF ENERGY: -341.3840944117161484 [AU]
(...)
Executing xmrcc...

**********************************************************************
MRCC( 2 ) calculation

Allocation of 100.0 Mbytes of memory...
Number of spinorbitals: 52
Number of alpha electrons: 6
Number of beta electrons: 4
Number of active particles: 6
Number of active holes: 8
Spin multiplicity: 2
z-component of spin: 1.0
Spatial symmetry: 2
Convergence criterion: 1.0E-08
Construction of occupation graphs...
Number of 0 -fold excitations: 1
Number of 1 -fold excitations: 70
Number of 2 -fold excitations: 3874
Number of 3 -fold excitations: 42209
Number of 4 -fold excitations: 158753
Number of 5 -fold excitations: 234881
Number of 6 -fold excitations: 141042
Number of 7 -fold excitations: 32044
Number of 8 -fold excitations: 2072
Total number of configurations: 614946
Calculation of coupling coefficients...
Length of intermediate file (Mbytes): 801.4
(...)
************************ 2020-11-26 16:06:18 *************************
Executing mrcc...

**********************************************************************
MRCCSD calculation


OpenMP parallel version is running.
Number of CPU cores: 3
Allocation of 15.4 Mbytes of memory...
Number of spinorbitals: 52
Number of alpha electrons: 6
Number of beta electrons: 4
Number of active particles: 6
Number of active holes: 8
Spin multiplicity: 2
z-component of spin: 1.0
Spatial symmetry: 2
Convergence criterion: 1.0E-08
Construction of occupation graphs...
Number of 0-fold excitations: 1
Number of 1-fold excitations: 70
Number of 2-fold excitations: 3874
Number of 3-fold excitations: 42209
Number of 4-fold excitations: 158753
Number of 5-fold excitations: 234881
Number of 6-fold excitations: 141042
Number of 7-fold excitations: 32044
Number of 8-fold excitations: 2072
Total number of determinants: 614946
Calculation of coupling coefficients...
Initial cluster amplitudes are generated.
Length of intermediate file (Mbytes): 801.4
Reading integral list from unit 55...
Sorting integrals...
Calculation of MP denominators...

Starting CC iteration...
======================================================================
Norm of residual vector: 1.09511714
CPU time [min]: 10.880 Wall time [min]: 3.687

Iteration 1 CC energy: -341.55866520 Energy decrease: 0.22766180
======================================================================
Norm of residual vector: 0.24381998
CPU time [min]: 21.647 Wall time [min]: 7.283

Iteration 2 CC energy: -341.54463411 Energy decrease: 0.01403110
======================================================================
Norm of residual vector: 0.09237414
CPU time [min]: 32.764 Wall time [min]: 10.997

Iteration 3 CC energy: -341.55602682 Energy decrease: 0.01139271
======================================================================
Norm of residual vector: 0.05825643
CPU time [min]: 43.839 Wall time [min]: 14.697

Iteration 4 CC energy: -341.56199167 Energy decrease: 0.00596485
======================================================================
Norm of residual vector: 0.04492909
CPU time [min]: 54.916 Wall time [min]: 18.401

Iteration 5 CC energy: -341.56494312 Energy decrease: 0.00295146
======================================================================


The list of files in the directory, before starting the B computations
CCDENSITIES
COORD.xyz
DAO
EXIT
FOCK
fort.10
fort.11
fort.12
fort.13
fort.131
fort.14
fort.16
fort.17
fort.18
fort.19
fort.20
fort.22
fort.23
fort.24
fort.28
fort.30
fort.55
fort.56
fort.78
fort.87
iface
KEYWD
MINP
MOCOEF
MOLDEN
MOLDEN.perm
OCCUP
OEINT
pids
PRINT
SCFDENSITIES
SCFDENSITIES.Al.cc-pvdz.0
SCFDENSITIES.F.cc-pvdz.0
SCHOL
SROOT
SYMTRA
TEDAT
TEINT
trust_region_data
VARS


The output generated by job B:
(...)
Release date: February 22, 2020
(...)
Input file:
(...)
basis=cc-pVDZ
calc=CCSD
ccprog=mrcc
scftype=mcscf
rest=1
occ=7,3,2,0/6,2,2,0
docc=5,1,1,0
mact=3,2,2,0
mem=15123MB
mult=3
charge=0
core=6
cctol=8
ccmaxit=3
maxmicroit=444
scfmaxit=222
verbosity=3

unit=bohr
geom
Al
F 1 R

R=3.1
(...)
Warning! scfalg has been set to disk!
Keywords:

active=none
agrid=ld0006-ld0590
basis=cc-pvdz
basis_sm=none
basopt=off
bfbasis=none
bfgsmem=10
bfgstol=1.0d-3
bpcompo=0.985
bpcompv=0.98
bpdfo=0.985
bpocc=0.985
bppdo=0.985
bppdv=0.98
bpedo=0.985
bpedv=0.98
calc=ccsd
ccmaxit=3
ccprog=mrcc
ccsdalg=disk
ccsdmkl=seq
ccsdrest=off
ccsdthreads=2
cctol=8
charge=0
cialg=disk
ciguess=off
cmpgrp=auto
core=6
corembed=off
csapprox=off
dboc=off
dendec=cholesky
dens=0
dfalg=invsqrt
dfbasis_cor=none
dfbasis_scf=none
dfintran=ovirt
dft=off
dhexc=adc(2)
diag=david
docc=5,1,1,0
domrad=10.d0
drpaalg=fit
dual=off
ecp=auto
edisp=off
edisp_embed=off
embed=off
epert=none
eps=0.975
excrad=0.d0
excrad_fin=0.000000000D+00
fmm=off
fmmord=8
freq=off
gauss=spher
geom=zmat
gopt=off
ghost=none
gtol=7
grdens=off
grtol=10
hamilton=dc
iface=none
intalg=auto
itol=12
laptol=1.d-2
lccoporder=trffirst
lcorthr=normal
lccrest=off
lmp2dens=on
lnoepso=0.d0
lnoepsv=1e-6
localcc=off
localcorrsymm=off
locintrf=disk
mact=3,2,2,0
maxact=off
maxdim=100
maxex=0
maxmicroit=444
mcscfiguess=hf
mem=15123mb
molden=on
mpitasks=1
mulmet=0
mult=3
nacto=0
nactv=0
naf_cor=off
naf_scf=off
nafalg=albe
naftyp=
nchol=auto
ndeps=1e-3
nstate=1
nsing=0
ntrip=0
occ=7,3,2,0/6,2,2,0
oniom=off
oniom_eechg=off
oniom_pcm=off
optalg=
optmaxit=50
optetol=1e-6
optex= 0
optgtol=1e-4
optstol=1e-3
orblocc=off
orbloce=off
orbloco=off
orblocv=off
orblocguess=cholesky
osveps=1e-3
ovirt=off
ovltol=1e-7
ovosnorb=80.0
pcm=off
popul=off
pressure=100000
pssp=off
ptfreq=0.0
ptthreads=2
qscf=aughessg
qmmm=off
redcost_exc=off
redcost_tddft=off
refdet=none
rest=1
rgrid=log3
rohfcore=semicanonical
rohftype=standard
scfalg=disk
scfdamp=off
scfdiis=on
scfdiis_end=222
scfdiis_start=1
scfdiis_step=1
scfdtol=9
scfext=10
scfguessdens=
scfiguess=sad
scflshift=off
scfmaxit=222
scftype=mcscf
scftol=8
scspe=1.d0
scsph=1.d0
scsps=1.2d0
scspt=0.33333333333333
scspv=1.d0
spairtol=1e-4
subminp=top
symm=
talg=occ
temp=298.15
test=off
theodore=off
tlmo=0.999
tpao=0.94
tprint=off
uncontract=off
unit=bohr
usedisk=2
verbosity=3
wpairtol=0.100000000E-05
(...)
************************ 2020-11-26 16:26:06 *************************
Executing xmrcc...

**********************************************************************
MRCC( 2 ) calculation

Allocation of 100.0 Mbytes of memory...
Number of spinorbitals: 52
Number of alpha electrons: 6
Number of beta electrons: 4
Number of active particles: 6
Number of active holes: 8
Spin multiplicity: 2
z-component of spin: 1.0
Spatial symmetry: 2
Convergence criterion: 1.0E-08
Construction of occupation graphs...
Number of 0 -fold excitations: 1
Number of 1 -fold excitations: 70
Number of 2 -fold excitations: 3874
Number of 3 -fold excitations: 42209
Number of 4 -fold excitations: 158753
Number of 5 -fold excitations: 234881
Number of 6 -fold excitations: 141042
Number of 7 -fold excitations: 32044
Number of 8 -fold excitations: 2072
Total number of configurations: 614946
Calculation of coupling coefficients...
Length of intermediate file (Mbytes): 801.4
(...)
**********************************************************************
MRCCSD calculation

OpenMP parallel version is running.
Number of CPU cores: 3
Allocation of 15.4 Mbytes of memory...
Number of spinorbitals: 52
Number of alpha electrons: 6
Number of beta electrons: 4
Number of active particles: 6
Number of active holes: 8
Spin multiplicity: 2
z-component of spin: 1.0
Spatial symmetry: 2
Convergence criterion: 1.0E-08
Construction of occupation graphs...
Number of 0-fold excitations: 1
Number of 1-fold excitations: 70
Number of 2-fold excitations: 3874
Number of 3-fold excitations: 42209
Number of 4-fold excitations: 158753
Number of 5-fold excitations: 234881
Number of 6-fold excitations: 141042
Number of 7-fold excitations: 32044
Number of 8-fold excitations: 2072
Total number of determinants: 614946
Calculation of coupling coefficients...
Initial cluster amplitudes are read from unit 16.
Length of intermediate file (Mbytes): 801.4
Reading integral list from unit 55...
Sorting integrals...
Calculation of MP denominators...

Starting CC iteration...
======================================================================
Norm of residual vector: 3.73660894
CPU time [min]: 10.923 Wall time [min]: 3.702

Iteration 1 CC energy: -341.55848606 Energy decrease: 0.23013121
======================================================================
Norm of residual vector: 1.20547221
CPU time [min]: 21.791 Wall time [min]: 7.332

Iteration 2 CC energy: -341.54470338 Energy decrease: 0.01378268
======================================================================
Norm of residual vector: 0.30601013
CPU time [min]: 32.940 Wall time [min]: 11.055

Iteration 3 CC energy: -341.55468264 Energy decrease: 0.00997926
======================================================================
Norm of residual vector: 0.10990047
CPU time [min]: 44.102 Wall time [min]: 14.787

Iteration 4 CC energy: -341.55918902 Energy decrease: 0.00450637
======================================================================
Norm of residual vector: 0.05843045
CPU time [min]: 55.275 Wall time [min]: 18.519

Iteration 5 CC energy: -341.56237227 Energy decrease: 0.00318326
======================================================================





I use the newest version of MRCC on the Linux cluster. I use MRCC as a standalone program. I installed the binary version and included patches for ovirt, mrcc, integ, dmrcc, scf.

What you did is the right way to restart the calculation. Probably the file fort.16, which is needed for the restart, was incomplete when the process was killed or the file got damaged.

Dear Mihaly,

thank you very much for your response. I guess that file fort.16 should be undamaged if MRCC stops normally (even if amplitudes are not fully converged). I estimated that I have time to complete about 15-20 iterations (I can not obtain more time for one job, due to the strict rules on the cluster). Unfortunately, when I set a small number of iterations (using ccmaxit), the MRCC do 50 iterations anyway. Do you have any suggestions?

Best Regards,
Marcin Gronowski

Dear Marcin,
You should edit file MRCCCOMMON, set the value of variable maxit, and recompile the code.

Dear Mihaly,

thank you very much. I ran into another problem with these MR-CCSD computations. It seems that there is a lack of convergence at the MR-CCSD level. The problem appears only with the 2020 version of MRCC. I have old outputs from the test with 2019 versions (the inputs are not exactly the same, but the differences are probably irrelevant). Computations, that employed 2019 versions converged (energy -341.441171767495 Hartree), whereas for 2020 energy is close to -341.056 Hartree. I see also a difference in MCSCF energy (version 2020: -341.2717335253845476; version 2019: -341.2714618579464059). Below, I present the part of the outputs from both versions of MRCC.

Best wishes,
Marcin



MRCC-2020
(...)
basis=aug-cc-pVDZ
calc=CCSD
ccprog=mrcc
scftype=mcscf

occ=7,3,2,0/6,2,2,0
docc=5,1,1,0
mact=3,2,2,0
mem=15123MB
mult=3
charge=0
core=6
cctol=8
scftol=8
ccmaxit=60
maxmicroit=444
scfmaxit=222
verbosity=3

unit=bohr
geom
Al
F 1 R

R=10
(...)
MRCCSD calculation

OpenMP parallel version is running.
Number of CPU cores: 3
Allocation of 121.8 Mbytes of memory...
Number of spinorbitals: 88
Number of alpha electrons: 6
Number of beta electrons: 4
Number of active particles: 6
Number of active holes: 8
Spin multiplicity: 2
z-component of spin: 1.0
Spatial symmetry: 2
Convergence criterion: 1.0E-08
Construction of occupation graphs...
Number of 0-fold excitations: 1
Number of 1-fold excitations: 112
Number of 2-fold excitations: 13249
Number of 3-fold excitations: 160020
Number of 4-fold excitations: 623325
Number of 5-fold excitations: 938035
Number of 6-fold excitations: 571968
Number of 7-fold excitations: 132164
Number of 8-fold excitations: 8623
Total number of determinants: 2447497
Calculation of coupling coefficients...
Initial cluster amplitudes are generated.
Length of intermediate file (Mbytes): 3227.5
Reading integral list from unit 55...
Sorting integrals...
Calculation of MP denominators...

Starting CC iteration...
======================================================================
Norm of residual vector: 0.50848976
CPU time [min]: 36.875 Wall time [min]: 13.035

Iteration 1 CC energy: -340.99739852 Energy decrease: 0.07193064
(...)
Iteration 38 CC energy: -341.05612967 Energy decrease: 0.00022098
======================================================================
Norm of residual vector: 0.01606625
CPU time [min]: 1520.260 Wall time [min]: 533.573

Iteration 39 CC energy: -341.05612600 Energy decrease: 0.00000366
======================================================================
Norm of residual vector: 0.01602809
CPU time [min]: 1559.821 Wall time [min]: 547.417

Iteration 40 CC energy: -341.05618004 Energy decrease: 0.00005404
======================================================================
Norm of residual vector: 0.01614671
CPU time [min]: 1598.822 Wall time [min]: 561.106

Iteration 41 CC energy: -341.05619507 Energy decrease: 0.00001504
======================================================================
Norm of residual vector: 0.01589222
CPU time [min]: 1637.728 Wall time [min]: 574.756

Iteration 42 CC energy: -341.05604530 Energy decrease: 0.00014978
======================================================================
Norm of residual vector: 0.01528578
CPU time [min]: 1677.380 Wall time [min]: 588.667

Iteration 43 CC energy: -341.05610945 Energy decrease: 0.00006415
======================================================================


MRCC-2019
(...)
basis=aug-cc-pVDZ
calc=CCSD
ccprog=mrcc
scftype=mcscf

occ=7,3,2,0/6,2,2,0
docc=5,1,1,0
mact=3,2,2,0
mem=15123MB
mult=3
charge=0
core=6
cctol=8
maxmicroit=444
qscf=AugHess

unit=bohr
geom
Al
F 1 R

R=10
(...)
Iteration 44 CC energy: -341.44117177 Energy decrease: 1.8639E-10
======================================================================
Norm of residual vector: 0.00000004
CPU time [min]: 1777.140 Wall time [min]: 453.815

Iteration 45 CC energy: -341.44117177 Energy decrease: 1.6553E-10
======================================================================
Norm of residual vector: 0.00000003
CPU time [min]: 1817.090 Wall time [min]: 464.021

Iteration 46 CC energy: -341.44117177 Energy decrease: 5.7071E-11
======================================================================
Norm of residual vector: 0.00000002
CPU time [min]: 1857.070 Wall time [min]: 474.233

Iteration 47 CC energy: -341.44117177 Energy decrease: 7.6227E-11
======================================================================
Norm of residual vector: 0.00000001
CPU time [min]: 1897.039 Wall time [min]: 484.444

Iteration 48 CC energy: -341.44117177 Energy decrease: 1.3983E-11
======================================================================
Norm of residual vector: 0.00000001
CPU time [min]: 1936.495 Wall time [min]: 494.526

Iteration 49 CC energy: -341.44117177 Energy decrease: 3.4731E-11
======================================================================

Iteration has converged in 49 steps.

Final results:
Total MRCCSD energy [au]: -341.441171767495