Keywords ignored!

Dear Mihaly,
Further to the message below,
I've added a couple of other examples from over the years.

out.1249524.txt is the same Li2 triplet state in aCV7Z instead of aCV6Z. The residual norm is huge, but interestingly it's quite ok in the first iteration, so maybe it's a problem with the integrals!

out.1377525.txt is not doing CCSD this time, but just CISD for H2. I tried both with the default CI, and with diag=olsen. In both cases the norm of the residual vector is huge in the first iteration.




I've pasted the 7Z basis set for Li and the 8Z basis set for He below:

LI:aug-cc-pCV7Zi
k,Li,0.3320,0.1719

14
0 0 1 1 2 2 3 3 4 4 5 5 6 6
9 6 8 6 7 5 6 4 5 3 4 2 3 1
19 6 11 6 7 5 6 4 5 3 4 2 3 1

245172. 35782.1 7999.22 2229.05 715.097
254.300 98.2097 40.4968 17.5830 7.94333
3.68942 1.74673 0.839107 0.399736 0.120210
0.0680452 0.0348770 0.0174420 0.0054

0.00000130 0.00000020 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00001044 0.00000163 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00005612 0.00000877 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00024246 0.00003788 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00090369 0.00014136 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00299226 0.00046826 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00894604 0.00140646 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.02436390 0.00385493 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.05971695 0.00962266 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.12761085 0.02118297 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.22760455 0.04028382 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.31591578 0.06437613 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0

72.0571 39.6354 21.8016 11.9921 6.5963 3.6283

1.0 0.0 0.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 0.0
0.0 0.0 0.0 0.0 1.0 0.0
0.0 0.0 0.0 0.0 0.0 1.0

57.1550 13.5136 4.32567 1.58025 0.610225
0.250245 0.110681 0.0519838 0.0253946 0.0124506
0.0039

0.00008775 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00073321 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00353008 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.01153326 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00000000 1.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00000000 0.0 1.0 0.0 0.0 0.0 0.0 0.0
0.00000000 0.0 0.0 1.0 0.0 0.0 0.0 0.0
0.00000000 0.0 0.0 0.0 1.0 0.0 0.0 0.0
0.00000000 0.0 0.0 0.0 0.0 1.0 0.0 0.0
0.00000000 0.0 0.0 0.0 0.0 0.0 1.0 0.0
0.00000000 0.0 0.0 0.0 0.0 0.0 0.0 1.0

78.7240 41.3309 21.6991 11.3923 5.9811 3.1401

1.0 0.0 0.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 0.0
0.0 0.0 0.0 0.0 1.0 0.0
0.0 0.0 0.0 0.0 0.0 1.0

0.9900 0.5445 0.2995 0.1648 0.0906 0.0498 0.0274

1.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 1.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 1.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 1.0

51.9421 25.1052 12.1341 5.8648 2.8346

1.0 0.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0
0.0 0.0 0.0 0.0 1.0

0.8899 0.4895 0.2692 0.1481 0.0815 0.0448

1.0 0.0 0.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 0.0
0.0 0.0 0.0 0.0 1.0 0.0
0.0 0.0 0.0 0.0 0.0 1.0

36.4855,16.4805,7.4442,3.3626

1.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0
0.0 0.0 1.0 0.0
0.0 0.0 0.0 1.0

0.00008775 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00073321 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00353008 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.01153326 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00000000 1.0 0.0 0.0 0.0 0.0 0.0 0.0
0.00000000 0.0 1.0 0.0 0.0 0.0 0.0 0.0
0.00000000 0.0 0.0 1.0 0.0 0.0 0.0 0.0
0.00000000 0.0 0.0 0.0 1.0 0.0 0.0 0.0
0.00000000 0.0 0.0 0.0 0.0 1.0 0.0 0.0
0.00000000 0.0 0.0 0.0 0.0 0.0 1.0 0.0
0.00000000 0.0 0.0 0.0 0.0 0.0 0.0 1.0

78.7240 41.3309 21.6991 11.3923 5.9811 3.1401

1.0 0.0 0.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 0.0
0.0 0.0 0.0 0.0 1.0 0.0
0.0 0.0 0.0 0.0 0.0 1.0

0.9900 0.5445 0.2995 0.1648 0.0906 0.0498 0.0274

1.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 1.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 1.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 1.0

51.9421 25.1052 12.1341 5.8648 2.8346

1.0 0.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0
0.0 0.0 0.0 0.0 1.0

0.8899 0.4895 0.2692 0.1481 0.0815 0.0448

1.0 0.0 0.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0 0.0
0.0 0.0 0.0 0.0 1.0 0.0
0.0 0.0 0.0 0.0 0.0 1.0

36.4855,16.4805,7.4442,3.3626

1.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0
0.0 0.0 1.0 0.0
0.0 0.0 0.0 1.0

0.6957 0.3827 0.2105 0.1158 0.0639

1.0 0.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0
0.0 0.0 0.0 0.0 1.0

25.5554 10.4854 4.3022

1.0 0.0 0.0
0.0 1.0 0.0
0.0 0.0 1.0

0.4960 0.2728 0.1501 0.0825

1.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0
0.0 0.0 1.0 0.0
0.0 0.0 0.0 1.0

18.4523 6.8097

1.0 0.0
0.0 1.0

0.3922 0.2157 0.1187

1.0 0.0 0.0
0.0 1.0 0.0
0.0 0.0 1.0

14.8820

1.0

He:Hellmann8Z
HELLMANN

7
0 1 2 3 4 5 6
10 9 8 7 6 5 4
16 9 8 7 6 5 4

69636.57620 10421.3753 2369.7889 670.714 218.7325
78.97270 30.7962000 12.748300 5.52480 2.471400
1.12470 0.51670000 0.2393000 0.10940 0.03534
0.01142

0.00000212 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000
0.00001649 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000
0.00008683 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000
0.00036718 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000
0.00133582 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000
0.00433716 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000
0.01277297 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000
0.03409673 1.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000
0.08050425 0.0000000 1.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000
0.16153434 0.0000000 0.0000000 1.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000
0.26378403 0.0000000 0.0000000 0.0000000 1.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000
0.32460548 0.0000000 0.0000000 0.0000000 0.0000000 1.0000000 0.0000000
0.0000000 0.0000000 0.0000000
0.23597951 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 1.0000000
0.0000000 0.0000000 0.0000000
0.05426079 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
1.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 1.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 1.0000000

28.2150 13.2630 6.2350 2.9310 1.3780
0.6476 0.3044 0.09832 0.03176

1.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000
0.0000000 1.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000
0.0000000 0.0000000 1.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000
0.0000000 0.0000000 0.0000000 1.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 1.0000000 0.0000000 0.0000000
0.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 1.0000000 0.0000000
0.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 1.0000000
0.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
1.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 1.0000000

23.6240 11.2400 5.3480 2.5440 1.2110
0.5759 0.1860 0.06007

1.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000
0.0000000 1.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000
0.0000000 0.0000000 1.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000
0.0000000 0.0000000 0.0000000 1.0000000 0.0000000 0.0000000 0.0000000
0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 1.0000000 0.0000000 0.0000000
0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 1.0000000 0.0000000
0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 1.0000000
0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
1.0000000

19.4870 9.1590 4.3040 2.0230 0.9508
0.3071 0.09919

1.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 1.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 1.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000 1.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 1.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 1.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 1.0000000

15.405 7.007 3.187 1.450 0.4684
0.1513

1.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 1.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 1.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000 1.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 1.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 1.0000000

11.787 5.054 2.167 0.6999 0.2261

1.0000000 0.0000000 0.0000000 0.0000000 0.0000000
0.0000000 1.0000000 0.0000000 0.0000000 0.0000000
0.0000000 0.0000000 1.0000000 0.0000000 0.0000000
0.0000000 0.0000000 0.0000000 1.0000000 0.0000000
0.0000000 0.0000000 0.0000000 0.0000000 1.0000000

8.605 3.336 1.078 0.3483

1.0000000 0.0000000 0.0000000 0.0000000
0.0000000 1.0000000 0.0000000 0.0000000
0.0000000 0.0000000 1.0000000 0.0000000
0.0000000 0.0000000 0.0000000 1.0000000

Dear Mihaly,
In your last reply you asked for me to attach the output file, which I attached in my last post almost 2 weeks ago.

I think there is definitely something going wrong when the basis set gets big, because I attached three different examples (CCSD in Li2 aCV6Z, CCSD in Li2 aCV7Z, and CISD in He2 d-aug-cc-pV8Z).

I've now got the huge residuals after running in stand-alone MRCC with much tighter convergence criteria:

and the SCF definitely did converge:

Next I can try with CFOUR or MOLPRO doing the integrals, but my guess is that it won't help because I'm quite sure the MRCC integrals are ok (especially with itol=18 and scftol=13 converged).

Perhaps the only solution would be to start with an MP2 guess for the CCSD ?
This works for MOLPRO (where CCSD converges easily for not only aC6Z, but also aCV7Z too).

For ROHF, how is the initial guess for CCSD chosen in MRCC?

EDIT: Actually the Norm is only 1.52 in the first iteration, so the initial guess might not be the problem (and in the He2 example in my previous reply, it was CISD that struggled, not CCSD):

Therefore, maybe it's the integrals after all, but with itol=18 and convergence of SCF with scftol=13, it's hard for me to guess what's going wrong. Do you have any insights Mihaly?

With best wishes!
Nike

Dear Nike,
The AO integrals must be correct since there is no problem with the SCF. However, the CC code uses transformed MO integrals. Thus there may be a bug in the integral transformation. Please try with the Cfour or Molpro interface because these pass over directly the MO integrals to mrcc, and the integral transformation cannot cause any problem.
I have started test this, but it is very slow.

Dear Mihaly,
Thanks for your reply!

I'm waiting for another large-RAM node to free-up so that I can run it with the CFOUR interface. The MOLPRO interface has this bug: www.molpro.net/pipermail/molpro-user/2015-January/006312.html which makes it impossible to see the output until the job finishes (which could be after a very long time). It seems I have to re-compile MOLPRO with Jan Martin's fix for this, but the MOLPRO on my cluster was compiled by someone else who bought the license several years ago so I will have to look into it.

I agree that the MO integrals is probably the problem. Is it not possible to implement an AO-based algorithm like what MOLPRO uses, or what CFOUR uses with the keyword ABCDTYPE=AOBASIS ?

The MRCC AO -> MO transformation takes sometimes 7 days to do (for Li2 in a 7-zeta basis set for example), and it's very heavy I/O on the disks, which often crashes the job. Sometimes I wait 7 days for the AO -> MO transformation even when the CCSDT only takes 1-2 days. This is because even though the formal scaling of CCSDT is worse than the O(N^4) integral sorting algorithm, I/O can be orders of magnitude slower than computation.

MOLPRO and CFOUR can do CCSD(T) from beginning to end (including the integrals) on Li2/aug-cc-pCV7Z in less than 2 hours, but in MRCC it takes 7 days for the AO -> MO transformation.

I gave the timing for the Li atom in 7Z here on the MRCC forum: www.mrcc.hu/index.php/forum/running-mrcc...o-transformation#429 . MRCC took 3 hours for the AO -> MO but MOLPRO took 127 seconds.

Alternatively, perhaps using the Parallel-HDF5 library can be used for the integral sorting routine to speed up the I/O by orders of magnitude. Currently ovirt seems to be just using 1 core, is it true?

With best wishes!
Nike

Dear Mihaly,
Did you have any luck with the tests you started 2 days ago?
- I tried doing CCSDT(Q) with MRCC using CFOUR to calculate the SCF and AO->MO transformation.
- It crashed a few times due to disk space issues. Finally I got the fort.55 written using a different disk. For aCV6Z fort.55 is 755GB and for aCV7Z fort.55 is 2.5TB. In both cases I got the same seg fault error:

The final thing I can think of is using stand-alone MRCC with UHF, since this way MP2 is calculated and maybe CCSD will start with a better initial guess. However I think the initial guess was not the problem, because the residual norm started at around 1.4 and then jumped to 1000, meaning that maybe there's something wrong with the MO integrals or with the coupled cluster at some point (maybe in the factorized equation generation or translation?).

I'll also try stand-alone MRCC with iface=cfour to trick it into reading the fort.55 file from CFOUR's integrals, but a 2.5TB fort.55 will not be quick to read if the reading is done with one core. I would suggest using the parallel-HDF5 library which would make it possible to read 2.5TB in less than 2 minutes with the use of many cores.

With best wishes!
Nike

Dear Mihaly,
In your last comment you said "I have started test this, but it is very slow." has there been any success?

I have determined that the issue is probably not the initial guess, since the same problem is now happening for a closed shell system (LiH) where MP2 should be used for the initial guess (even though it won't print the MP2 until after the CCSD is done, I assume MP2 is calculated before the CCSD, in order to be used as an initial guess).

Below I show that for aCV6Z, the residual norms are small and are getting smaller:

But for aCV7Z the residuals rapidly become huge:

For aCV7Z, SCF converged in 19 steps, even with scftol=12.
I also have itol=18, so the integrals should be good.


One thing in common for all cases where the norms get huge, is that the number of spin orbitals is over 1000. For the case where the norms were small, there was only about 800 spin orbitals. Could the bug be due to going past ~1000 spin orbitals? At around spin orbitals we had a bug in the FCIQMC code due to using int32 instead of int64, which affected the array indexing. This was fixed by using int64, but I'm not sure if it's the same problem in MRCC.

I think we can conclude:
1) it happens at around 1000 spin orbitals,
2) it doesn't depend on the initial guess, because the first iteration has a small norm, but the norm grows in the following iterations
3) The residuals grow large with both RHF and ROHF
4) The residuals grow larger with hand-coded CCSD, automatic-coded CCSD, and also CISD
5) SCF converged in every case, so the AO integrals are fine
6) Residuals are getting big eveb with itol=18, and scftol=12, so AO integrals are fine
7) There seems to be something wrong with the AO to MO transformation for all cases with > 1000 spin orbitals, but this problem does NOT exist when there's ~800 spin orbitals.