Newton casscf hcc (issue #701) (pyscf/pyscf#1869)

This merge request fixes the errors in mcscf.newton_casscf.gen_g_hop in the calculation of the CI vector second derivative of the (state-averaged) energy [here](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L296-L313) as described in issue #701. The newton_casscf algorithm does not directly use the CI trial vector computed by the gradient-descent step using this function to update the result CI vector without first running the actual FCI kernel, and the coupling between the orbital and CI degrees of freedom likewise used a robust expression that had the effect of canceling the erroneous contribution, so these errors did not lead to erroneous results in the sense that converged MC-SCF calculations were always correctly converged. This fix should therefore not change the outcome of any calculations in which the MC-SCF iteration converges to the correct stationary point, and the only unittest changes are to meaningless arbitrary reference points. However, this fix does cause the function to finally _do what it says it does,_ which IMO is important. For completeness, and since my original issue page used a third-party plugin for equations that may not render correctly for everyone, I now repeat the derivation using the native markdown latex support.

The first and second derivatives of an expectation value such as the with respect to some sort of unitary transformation are obtained by expressing the unitary operator as the exponential of the corresponding antihermitian unitary group generators,

$$
E = \langle 0|e^{-\hat{G}} \hat{H} e^{\hat{G}}|0\rangle
$$

For the CI vector part of an MC-SCF wave function in pyscf.mcscf.newton_casscf.py, the Hamiltonian-vector product is functionally abstracted, so we don't need to think too hard about the form of $\hat{G}$; we can just use the basic bra-ket language and the definition of antihermiticity:

$$
\hat{G} = \sum_{I} g_I (|I\rangle\langle0| - |0\rangle\langle I|) = |\vec{g}\rangle\langle 0| - |0\rangle\langle\vec{g}|
$$

It follows after some straightforward tedium that the first and second derivatives with respect to the generators are

$$
\frac{\partial E}{\partial g_I} = \langle I|\hat{H} - E_0|0\rangle + \textrm{h.c.}
$$

$$
\frac{\partial^2 E}{\partial g_I\partial g_J} = \langle I|\hat{Q}(\hat{H}-E_0)\hat{Q}|J\rangle + \textrm{h.c.}
$$

with

$$
\hat{Q} = 1 - |0\rangle\langle 0| ; E_0 = \langle 0|\hat{H}|0\rangle
$$

Using that $\langle 0|\hat{H}-E_0|0\rangle=0$ and that for real wave functions $\langle I|0\rangle = \langle 0|I\rangle = c_I$, the CI vector for the keyframe state $|0\rangle$, the second derivative is expanded as

$$
\frac{\partial^2 E}{\partial g_I\partial g_J} = \langle I|\hat{H}-E_0|J\rangle -\frac{1}{2}(c_I\frac{\partial E}{\partial g_J} + \frac{\partial E}{\partial g_I}c_J) + \textrm{h.c.}
$$

Note that "+ h.c." in both the first and second derivatives are accounted for by factors of 2 [here](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L303) and [here](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L384) that are applied at the end of the calculation. This means that `gci` at [this](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L296C15-L296C15) point in the function stores $\frac{1}{2}\frac{\partial E}{\partial g_I}$. Hopefully it is now clear why the factor of 4 on [this](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L296C15-L296C15) line is incorrect and should be a factor of 2. For the other two erroneous factors [here](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L312-L313), we can determine the correct expression by treating the second derivative as an operator, $\langle I|\partial^2 E|J\rangle \equiv \frac{\partial^2 E}{\partial g_I\partial g_J}$:

$$
\partial^2 E|I\rangle = (\hat{H} - E_0)\ket{I} - (\hat{H} - E_0)\ket{0}\braket{0|I} - \ket{0}\braket{0|\hat{H}-E_0|I} + \mathrm{h.c.}
$$

Again, "+ h.c." is accounted for later in the function. Hopefully it is clear that factors of 2 simply should not be [here](https://github.com/pyscf/pyscf/blob/master/pyscf/mcscf/newton_casscf.py#L312-L313).

Is there a formal error in the H_cc sector of newton_casscf.gen_g_hop?

In differentiating with respect to an infinitesimal transformation of the CI vectors, we parameterize the energy as

with

The second derivative of the energy is

with

or, if we can forgive a bit of notational handwaving,

But what's programmed in newton_casscf.py appears to be

in both hop and hdiag. I would understand a factor of 2 convention difference (EDIT: actually all three of my second-derivative expressions up there should have a +h.c. that I dropped), but this isn't a factor of 2, unless I've gotten confused about what (H-E0) is. In fact, it spoils intermediate normalization (we should have <0|x>=0 at all times; the component of P along the trial vector |0> is undefined by construction):

```
import numpy
from pyscf import gto, scf, lib, fci
from pyscf.mcscf import newton_casscf, CASSCF, addons

mol = gto.Mole()
mol.verbose = lib.logger.DEBUG
mol.output = '/dev/null'
mol.atom = [
    ['H', ( 5.,-1.    , 1.   )],
    ['H', ( 0.,-5.    ,-2.   )],
    ['H', ( 4.,-0.5   ,-3.   )],
    ['H', ( 0.,-4.5   ,-1.   )],
    ['H', ( 3.,-0.5   ,-0.   )],
    ['H', ( 0.,-3.    ,-1.   )],
    ['H', ( 2.,-2.5   , 0.   )],
    ['H', ( 1., 1.    , 3.   )],
]
mol.basis = 'sto-3g'
mol.build()
mf = scf.RHF(mol)
mf.max_cycle = 3
mf.kernel()
mc = newton_casscf.CASSCF(mf, 4, 4)
mc.fcisolver = fci.direct_spin1.FCI(mol)
mc.kernel()

numpy.random.seed(1)
mo = numpy.random.random(mf.mo_coeff.shape)
ci0 = numpy.random.random((6,6))
ci0/= numpy.linalg.norm(ci0)
gall, gop, hop, hdiag = newton_casscf.gen_g_hop(mc, mo, ci0, mc.ao2mo(mo))
x = numpy.random.random(gall.size)
hx = hop (x)
print ("|hx_ci| =",numpy.linalg.norm (hx[-36:]),"<0|hx> =", ci0.ravel ().dot (hx[-36:]))
```

yields

```
|hx_ci| = 73.30773999100957 <0|hx> = 4.13369273110702
```

I think this has minimal effects on the results of calculations because 1) the extract_rotation function forcibly normalizes the new output CI vector after every Newton-CG step, and the s10 business in the H_oc sector corrects for the effect of the spurious <0|x> component on the orbitals, so all this does is attenuate CI vector evolution almost like a trust radius, and 2) the coupled-perturbed solver for the state-average analytical gradient calculation that I added has a preconditioner that enforces <0|x> = 0 post hoc. But I wanted to ask about it to see if I was missing something, especially since changing it would make at least one test fail.

diff --git a/pyscf/fci/rdm.py b/pyscf/fci/rdm.py
index 58be01c4fe..284053c014 100644
--- a/pyscf/fci/rdm.py
+++ b/pyscf/fci/rdm.py
@@ -101,8 +101,8 @@ def make_rdm1_spin1(fname, cibra, ciket, norb, nelec, link_index=None):
         link_indexa, link_indexb = link_index
     na,nlinka = link_indexa.shape[:2]
     nb,nlinkb = link_indexb.shape[:2]
-    assert (cibra.size == na*nb)
-    assert (ciket.size == na*nb)
+    assert (cibra.size == na*nb), '{} {} {}'.format (cibra.size, na, nb)
+    assert (ciket.size == na*nb), '{} {} {}'.format (ciket.size, na, nb)
     rdm1 = numpy.empty((norb,norb))
     fn = getattr(librdm, fname)
     fn(rdm1.ctypes.data_as(ctypes.c_void_p),
diff --git a/pyscf/mcscf/newton_casscf.py b/pyscf/mcscf/newton_casscf.py
index 941def1e56..b6e4d9c352 100644
--- a/pyscf/mcscf/newton_casscf.py
+++ b/pyscf/mcscf/newton_casscf.py
@@ -293,7 +293,8 @@ def g_update(u, fcivec):
 #    h_diag[ncore:nocc,ncore:nocc] -= v_diag[:,ncore:nocc]*2
     h_diag = casscf.pack_uniq_var(h_diag)

-    hci_diag = [hd - ec - gc*c*4 for hd, ec, gc, c in zip (_Hdiag (h1cas_0, eri_cas), eci0, gci, ci0)]
+    # Fix intermediate normalization? - MRH 2023/09/15
+    hci_diag = [hd - ec - gc*c*2 for hd, ec, gc, c in zip (_Hdiag (h1cas_0, eri_cas), eci0, gci, ci0)]
     hci_diag = [h * w for h, w in zip (hci_diag, weights)]
     hdiag_all = numpy.hstack((h_diag*2, _pack_ci (hci_diag)*2))

@@ -308,9 +309,10 @@ def h_op(x):
         ci1 = _unpack_ci (x[ngorb:])

         # H_cc
+        # Fix intermediate normalization? - MRH 2023/09/15
         hci1 = [hc1 - c1 * ec0 for hc1, c1, ec0 in zip (_Hci (h1cas_0, eri_cas, ci1), ci1, eci0)]
-        hci1 = [hc1 - 2*(hc0 - c0*ec0)*c0.dot(c1) for hc1, hc0, c0, ec0, c1 in zip (hci1, hci0, ci0, eci0, ci1)]
-        hci1 = [hc1 - 2*c0*(hc0 - c0*ec0).dot(c1) for hc1, hc0, c0, ec0, c1 in zip (hci1, hci0, ci0, eci0, ci1)]
+        hci1 = [hc1 - (hc0 - c0*ec0)*c0.dot(c1) for hc1, hc0, c0, ec0, c1 in zip (hci1, hci0, ci0, eci0, ci1)]
+        hci1 = [hc1 - c0*(hc0 - c0*ec0).dot(c1) for hc1, hc0, c0, ec0, c1 in zip (hci1, hci0, ci0, eci0, ci1)]

         # H_co
         rc = x1[:,:ncore]

diff --git a/pyscf/mcscf/test/test_newton_casscf.py b/pyscf/mcscf/test/test_newton_casscf.py
index db4e7c15a7..f2d94c3e9a 100644
--- a/pyscf/mcscf/test/test_newton_casscf.py
+++ b/pyscf/mcscf/test/test_newton_casscf.py
@@ -88,11 +88,11 @@ def test_gen_g_hop(self):
         ci0/= numpy.linalg.norm(ci0)
         gall, gop, hop, hdiag = newton_casscf.gen_g_hop(mc, mo, ci0, mc.ao2mo(mo))
         self.assertAlmostEqual(lib.fp(gall), 21.288022525148595, 8)
-        self.assertAlmostEqual(lib.fp(hdiag), -4.6864640132374618, 8)
+        self.assertAlmostEqual(lib.fp(hdiag), -15.618395788969842, 8)
         x = numpy.random.random(gall.size)
         u, ci1 = newton_casscf.extract_rotation(mc, x, 1, ci0)
         self.assertAlmostEqual(lib.fp(gop(u, ci1)), -412.9441873541524, 8)
-        self.assertAlmostEqual(lib.fp(hop(x)), 73.358310983341198, 8)
+        self.assertAlmostEqual(lib.fp(hop(x)), 24.04569925660985, 8)

     def test_get_grad(self):
         # mc.e_tot may be converged to -3.6268060853430573
@@ -107,11 +107,11 @@ def test_sa_gen_g_hop(self):
         ci0 = list (ci0.reshape ((2,6,6)))
         gall, gop, hop, hdiag = newton_casscf.gen_g_hop(sa, mo, ci0, sa.ao2mo(mo))
         self.assertAlmostEqual(lib.fp(gall), 32.46973284682045, 8)
-        self.assertAlmostEqual(lib.fp(hdiag), -63.6527761153809, 8)
+        self.assertAlmostEqual(lib.fp(hdiag), -70.61862254321517, 8)
         x = numpy.random.random(gall.size)
         u, ci1 = newton_casscf.extract_rotation(sa, x, 1, ci0)
         self.assertAlmostEqual(lib.fp(gop(u, ci1)), -49.017079186126, 8)
-        self.assertAlmostEqual(lib.fp(hop(x)), 169.47893548740288, 8)
+        self.assertAlmostEqual(lib.fp(hop(x)), 136.2077988624156, 8)

     def test_sa_get_grad(self):
         self.assertAlmostEqual(sa.e_tot, -3.62638372957158, 7)


Assume 46 and 78 core configurations to be f-in-valence (pyscf/pyscf#1891)

This is a reasonable default and should fix  #1889. I added a test with `mf.get_init_guess()` to catch the error.

What should be the way to introduce another core configuration to the same element and number of electrons?
An `f_in_core`  attribute should be added somewhere.

Using ECP gives an error for Actinide elements

I am trying to do a HF/DFT calculation on a system involving a U atom and trying to invoke the effective core potential (ECP) 'crenbl' but that gives a numpy error:   LinAlgError('Last 2 dimensions of the array must be square'). The error goes away if ECP is turned off. The same thing happens when using other ECPs (eg. 'stuttgart') listed in the pyscf User Guide or on Basis set exchange for U and other Actinide elements. I am using pyscf v2.3.0 with numpy v1.26.0, scipy v1.11.3, h5py v3.9.0 and python v3.10.13 on Mac ARM architechture. The code is as follows:

```python
from pyscf import gto, scf, dft

mol = gto.Mole()
mol.atom='''
U    -0.00000    0.00000    0.00000
H     -1.70048   -1.32378   -0.00000
H      1.70048   -1.32378   -0.00000
'''
mol.basis={'U':'crenbl','H':'6-31++g**'}
mol.ecp={'U':'crenbl'}
mol.symmetry = True
mol.build()

mf = dft.ROKS(mol)
mf.xc='b3lyp,b3lyp'
mf.max_cycle = 60
mf.conv_tol = 1e-9
mf.kernel()
```

diff --git a/pyscf/gto/ecp.py b/pyscf/gto/ecp.py
index e686b88f26..d6a264318e 100644
--- a/pyscf/gto/ecp.py
+++ b/pyscf/gto/ecp.py
@@ -145,10 +145,12 @@ def core_configuration(nelec_core, atom_symbol=None):
     elements_4f = ELEMENTS[57:71]
     elements_5f = ELEMENTS[89:103]
     if atom_symbol in elements_4f:
-        for i in range(46, 60):
+        # TODO: fix La3+ and Ce4+ ECP46MWB f-in-core ECPs
+        for i in range(47, 59):
             conf_dic[i] = '4s3p2d1f'
     if atom_symbol in elements_5f:
-        for i in range(78, 92):
+        # TODO: fix Ac3+, Th4+, Pa5+ and U6+ ECP78MWB f-in-core ECPs
+        for i in range(79, 91):
             conf_dic[i] = '5s4p3d2f'
     if nelec_core not in conf_dic:
         raise RuntimeError('Core configuration for %d core electrons is not available.' % nelec_core)

diff --git a/pyscf/gto/test/test_ecp.py b/pyscf/gto/test/test_ecp.py
index f35d0e1f9e..d2341396f3 100644
--- a/pyscf/gto/test/test_ecp.py
+++ b/pyscf/gto/test/test_ecp.py
@@ -376,9 +376,19 @@ def test_ecp_f_in_core(self):
 2      5.3988000000      -63.6010500000
                     ''')}, charge=2, verbose=0)
         mf = scf.RHF(mol)
+        mf.get_init_guess()
         self.assertEqual(mol.ao_labels()[0], '0 Eu1 5s    ')
         self.assertAlmostEqual(lib.fp(mf.get_hcore()), 22.59028455662168)

+    def test_ecp_f_in_valence(self):
+        mol = gto.M(atom='U, 0.00, 0.00, 0.00',
+                    basis={'U': 'crenbl'}, ecp={'U': 'crenbl'},
+                    charge=3, spin=3, verbose=0)
+        mf = scf.ROHF(mol)
+        mf.get_init_guess()
+        self.assertEqual(mol.ao_labels()[40], '0 U 5f-3  ')
+        self.assertAlmostEqual(lib.fp(mf.get_hcore()), -55.38627201912257)
+

 if __name__ == '__main__':
     print("Full Tests for ECP")


Fix CASCI grad (pyscf/pyscf#1914)

ValueError when calculating CASCI gradients.

PySCF Version 2.0.1 and 2.4.0
## Minimal working example:
```
from pyscf import M
from pyscf.mcscf import CASCI

geom_str = """
    H                 -0.00021900   -0.20486000   -2.17721200
    H                 -0.00035900   -1.27718700   -2.17669400
    """

mol = M(
    atom = geom_str,
    basis='6-31G',
    charge=0,
    spin=0)

hf = mol.HF()
print(hf.kernel())
casci = CASCI(hf, 2, 2)
print(casci.kernel()[0])
grad = casci.nuc_grad_method()
grad.max_memory = 0
nuc_grad = grad.kernel()
print(nuc_grad)

```
## Error output
```
  File "XXX", line 35, in <module>
    nuc_grad = grad.kernel()
  File "/XXX/site-packages/pyscf/grad/casci.py", line 305, in kernel
    de = self.grad_elec(mo_coeff, ci, atmlst, log)
  File "/XXX/site-packages/pyscf/grad/casci.py", line 161, in grad_elec
    de[k,i] += numpy.einsum('ijkl,il,kj', eri1tmp, hf_dm1[p0:p1], zvec_ao[q0:q1])
  File "<__array_function__ internals>", line 6, in einsum
  File "/XXX/site-packages/numpy/core/einsumfunc.py", line 1361, in einsum
    return c_einsum(*operands, **kwargs)
ValueError: operands could not be broadcast together with remapped shapes [original->remapped]: (2,2,4,4)->(2,2,4,4) (2,4)->(2,newaxis,newaxis,4) (2,4)->(4,2,newaxis)
python-BaseException
```
## Analysis
The indices for the einsum in `grad_elec` line 161 seem to be incorrect.
`eri1tmp` is reshaped by
```
eri1tmp = eri1tmp.reshape(p1-p0,nf,nao,nao)
```
The shape of `hf_dm1[p0:p1]` is `(p1-p0, nao)` and the shape of `zvec_ao[q0:q1]` is `(nf, nao)`.
It seems the correct einsum index should be `'ijkl,il,jk'` instead of `'ijkl,il,kj'`.

The bug will cause a `ValueError` when calculating the gradient for large molecules or the max memory is set to a small value.
Note that the same bug (if it truly is) is also present in the code after `grad_elec` line 161.

diff --git a/pyscf/grad/casci.py b/pyscf/grad/casci.py
index 1d46856e69..ccc29af896 100644
--- a/pyscf/grad/casci.py
+++ b/pyscf/grad/casci.py
@@ -160,15 +160,15 @@ def fvind(x):
                 eri1tmp = eri1tmp.reshape(p1-p0,nf,nao,nao)
                 de[k,i] -= numpy.einsum('ijkl,ij,kl', eri1tmp, hf_dm1[p0:p1,q0:q1], zvec_ao) * 2
                 de[k,i] -= numpy.einsum('ijkl,kl,ij', eri1tmp, hf_dm1, zvec_ao[p0:p1,q0:q1]) * 2
-                de[k,i] += numpy.einsum('ijkl,il,kj', eri1tmp, hf_dm1[p0:p1], zvec_ao[q0:q1])
+                de[k,i] += numpy.einsum('ijkl,il,kj', eri1tmp, hf_dm1[p0:p1], zvec_ao[:,q0:q1])
                 de[k,i] += numpy.einsum('ijkl,jk,il', eri1tmp, hf_dm1[q0:q1], zvec_ao[p0:p1])

                 #:vhf1c, vhf1a = mc_grad.get_veff(mol, (dm_core, dm_cas))
                 #:de[k] += numpy.einsum('xij,ij->x', vhf1c[:,p0:p1], casci_dm1[p0:p1]) * 2
                 #:de[k] += numpy.einsum('xij,ij->x', vhf1a[:,p0:p1], dm_core[p0:p1]) * 2
-                de[k,i] -= numpy.einsum('ijkl,lk,ij', eri1tmp, dm_core[q0:q1], casci_dm1[p0:p1]) * 2
+                de[k,i] -= numpy.einsum('ijkl,lk,ij', eri1tmp, dm_core, casci_dm1[p0:p1,q0:q1]) * 2
                 de[k,i] += numpy.einsum('ijkl,jk,il', eri1tmp, dm_core[q0:q1], casci_dm1[p0:p1])
-                de[k,i] -= numpy.einsum('ijkl,lk,ij', eri1tmp, dm_cas[q0:q1], dm_core[p0:p1]) * 2
+                de[k,i] -= numpy.einsum('ijkl,lk,ij', eri1tmp, dm_cas, dm_core[p0:p1,q0:q1]) * 2
                 de[k,i] += numpy.einsum('ijkl,jk,il', eri1tmp, dm_cas[q0:q1], dm_core[p0:p1])
             eri1 = eri1tmp = None

diff --git a/pyscf/grad/test/test_casci.py b/pyscf/grad/test/test_casci.py
index 4fbcd96b32..b75d552b1b 100644
--- a/pyscf/grad/test/test_casci.py
+++ b/pyscf/grad/test/test_casci.py
@@ -459,12 +459,19 @@ def test_symmetrize(self):
         g = mol.RHF.run().CASCI(4, 4).run().Gradients().kernel()
         self.assertAlmostEqual(lib.fp(g), 0.11555543375018221, 6)

+    # issue 1909
+    def test_small_mem(self):
+        mol = gto.M(atom="""
+            H                 -0.00021900   -0.20486000   -2.17721200
+            H                 -0.00035900   -1.27718700   -2.17669400
+            """, basis='6-31G')
+        casci = mol.CASCI(2, 2).run()
+        grad = casci.nuc_grad_method()
+        grad.max_memory = 0
+        nuc_grad = grad.kernel()
+        self.assertAlmostEqual(lib.fp(nuc_grad), 0.09424065197659935, 7)
+

 if __name__ == "__main__":
     print("Tests for CASCI gradients")
-    #unittest.main()
-    if 1:
-        setUpModule()
-        mc = mcscf.CASCI(mf, 4, 4)
-        mc = mcscf.addons.state_average_mix_(mc, [mc.fcisolver, mc.fcisolver], (.5, .5))
-        gs = mc.nuc_grad_method().as_scanner()
+    unittest.main()


fix get_bands with k-point symmetry (pyscf/pyscf#1927)

scf.KUKS(cell, kpts).getbands() has problems with symmetry

Hello,

I am using the scf.KUKS object to accelerate my simulations. But when I want to plot the band diagram, it has a problem with the length of different arrays in the KNumInt.eval_rho2 function. ao_kpts has a length of 27 (3x3x3) while mo_coeff_kpts and mo_occ_kpts both have a length of 4 due to symmetry. Am I doing something wrong or is it a bug in the pySCF code?

```kmesh = [3, 3, 3]
kpts = cell.make_kpts(kmesh, space_group_symmetry=True, time_reversal_symmetry=True)

run_scf = True

if run_scf:
    PBE = scf.KUKS(cell, kpts)
    PBE = scf.addons.smearing_(PBE, sigma=.1, method='fermi')
    PBE.xc = 'PBE'
    PBE.verbose = 4
    PBE.init_guess_breaksym=False
    PBE.init_guess = "1e"
    PBE.chkfile = 'SrVO3_PBE.chk'
    PBE.run();

## Calculate PBE bands non-selfconsistently
run_bands = True

# Define high-symmetry k-paths:
hspoints = {
  "G": [0.0, 0.0, 0.0],
  "X": [0.5, 0.0, 0.0],
  "M": [0.5, 0.5, 0.0],
  "R": [0.5, 0.5, 0.5]}

G = hspoints['G']
X = hspoints['X']
M = hspoints['M']
R = hspoints['R']
kpath = [G, X, M, G, R, X]
band_kpts, kpath, sp_points = get_bandpath(kpath, cell.a, npoints=50);
# band_kpts = libkpts.make_kpts(cell, band_kpts, space_group_symmetry=True, time_reversal_symmetry=True)
# band_kpts = cell.get_scaled_kpts(band_kpts, kpts_in_ibz=True)
band_kpts = cell.get_abs_kpts(band_kpts);

print(band_kpts)

# nscf along high-symmetry path:
if run_bands:
    nscf = scf.KUKS(cell, kpts)
    nscf.__dict__.update(chkfile.load("SrVO3_PBE.chk", 'scf'))
    E_nk = nscf.get_bands(np.array(band_kpts), kpts = kpts)[0] #the error occurs at this line
    E_F = nscf.get_fermi()
```

diff --git a/pyscf/pbc/dft/krks_ksymm.py b/pyscf/pbc/dft/krks_ksymm.py
index 087c0241f1..fb15bf6f40 100644
--- a/pyscf/pbc/dft/krks_ksymm.py
+++ b/pyscf/pbc/dft/krks_ksymm.py
@@ -42,14 +42,14 @@ def get_veff(ks, cell=None, dm=None, dm_last=0, vhf_last=0, hermi=1,
     ground_state = (isinstance(dm, np.ndarray) and dm.ndim == 3 and
                     kpts_band is None)

-    if kpts_band is None: kpts_band = kpts.kpts_ibz
-    dm_bz = dm
-    if ground_state:
-        if len(dm) != kpts.nkpts_ibz:
-            raise RuntimeError("Number of input density matrices does not \
-                               match the number of IBZ kpts: %d vs %d."
-                               % (len(dm), kpts.nkpts_ibz))
-        dm_bz = kpts.transform_dm(dm)
+    if kpts_band is None:
+        kpts_band = kpts.kpts_ibz
+
+    if len(dm) != kpts.nkpts_ibz:
+        raise KeyError('Shape of the input density matrix does not '
+                       'match the number of IBZ k-points: '
+                       f'{len(dm)} vs {kpts.nkpts_ibz}.')
+    dm_bz = kpts.transform_dm(dm)

     hybrid = ni.libxc.is_hybrid_xc(ks.xc)

diff --git a/pyscf/pbc/dft/kuks_ksymm.py b/pyscf/pbc/dft/kuks_ksymm.py
index 465b2943fb..eb02e674e9 100644
--- a/pyscf/pbc/dft/kuks_ksymm.py
+++ b/pyscf/pbc/dft/kuks_ksymm.py
@@ -41,14 +41,14 @@ def get_veff(ks, cell=None, dm=None, dm_last=0, vhf_last=0, hermi=1,
     # ndim = 4 : dm.shape = ([alpha,beta], nkpts, nao, nao)
     ground_state = (dm.ndim == 4 and dm.shape[0] == 2 and kpts_band is None)

-    if kpts_band is None: kpts_band = kpts.kpts_ibz
-    dm_bz = dm
-    if ground_state:
-        if len(dm[0]) != kpts.nkpts_ibz:
-            raise RuntimeError("Number of input density matrices does not \
-                               match the number of IBZ kpts: %d vs %d."
-                               % (len(dm[0]), kpts.nkpts_ibz))
-        dm_bz = kpts.transform_dm(dm)
+    if kpts_band is None:
+        kpts_band = kpts.kpts_ibz
+
+    if len(dm[0]) != kpts.nkpts_ibz:
+        raise KeyError('Shape of the input density matrix does not '
+                       'match the number of IBZ k-points: '
+                       f'{len(dm[0])} vs {kpts.nkpts_ibz}.')
+    dm_bz = kpts.transform_dm(dm)

     hybrid = ni.libxc.is_hybrid_xc(ks.xc)

diff --git a/pyscf/pbc/dft/test/test_krks_ksym.py b/pyscf/pbc/dft/test/test_krks_ksym.py
index d8977722d6..6c7bd46c4a 100644
--- a/pyscf/pbc/dft/test/test_krks_ksym.py
+++ b/pyscf/pbc/dft/test/test_krks_ksym.py
@@ -19,8 +19,10 @@
 import unittest
 import numpy as np

+from pyscf.lib import finger
 from pyscf.pbc import gto as pbcgto
 from pyscf.pbc import scf as pscf
+from pyscf.pbc.scf import chkfile
 from pyscf.pbc.dft import krks, kuks, multigrid

 def make_primitive_cell(mesh):
@@ -294,6 +296,33 @@ def test_convert_from(self):
         kumf.kernel(np.asarray([dm,dm]) / 2.)
         self.assertAlmostEqual(kmf.e_tot, kumf.e_tot, 7)

+    def test_get_bands(self):
+        kpts = cell.make_kpts(nk,
+                              space_group_symmetry=True,
+                              time_reversal_symmetry=True)
+        kumf = pscf.KUKS(cell, kpts=kpts)
+        kumf.xc = 'lda'
+        kumf.chkfile = 'test_get_bands_ksymm.chk'
+        kumf.kernel()
+
+        band_kpts = np.array([[-0.205736, 0.308604, 0.308604],
+                              [-0.25717,  0.308604, 0.308604]])
+        kumf = pscf.KUKS(cell, kpts=kpts)
+        kumf.__dict__.update(chkfile.load('test_get_bands_ksymm.chk', 'scf'))
+        E_nk = kumf.get_bands(band_kpts, kpts=kpts)[0]
+        E_F = kumf.get_fermi()
+        self.assertAlmostEqual(finger(np.asarray(E_nk[0])), 0.5575755379561839, 6)
+        self.assertAlmostEqual(finger(np.asarray(E_nk[1])), 0.5575755379561839, 6)
+        self.assertAlmostEqual(E_F[0], 0.3093399745201863, 6)
+        self.assertAlmostEqual(E_F[1], 0.3093399745201863, 6)
+
+        kmf = pscf.KRKS(cell, kpts=kpts)
+        kmf = kmf.convert_from_(kumf)
+        E_nk = kmf.get_bands(band_kpts, kpts=kpts)[0]
+        E_F = kmf.get_fermi()
+        self.assertAlmostEqual(finger(np.asarray(E_nk)), 0.5575755379561839, 6)
+        self.assertAlmostEqual(E_F, 0.3093399745201863, 6)
+
 if __name__ == '__main__':
     print("Full Tests for DFT with k-point symmetry")
     unittest.main()


Fix decontract_basis function when basis are missed on certain atoms. (Fix #1711) (pyscf/pyscf#1943)

Choice of basis sets for multi-grid density fitting (NaCl)

Hi all,

Can someone please shed some light on why this simple test crashes? This is pretty much based on `examples/pbc/27-multigrid.py`, but I might easily still be doing something very stupid.

```
#!/usr/bin/env python

import numpy as np
from pyscf.pbc import df, dft, gto
from pyscf.pbc.dft import multigrid

cell = gto.Cell()
cell.atom = 'Cl 0 0 1; Na 0 1 0'
cell.a = np.eye(3) * 4.
cell.basis = 'gth-dzvp'
cell.pseudo = 'gth-pbe'
cell.verbose = 4
cell.build()

mf = dft.KRKS(cell)
mf.xc = 'pbe'
mf.with_df = multigrid.MultiGridFFTDF(cell)
mf.verbose = 4
mf.kernel()
```

I'm getting the following traceback:

```
******** PBC SCF flags ********
N kpts = 1
Exchange divergence treatment (exxdiv) = ewald
    madelung (= occupied orbital energy shift) = 0.3753582916854765
    Total energy shift due to Ewald probe charge = -1/2 * Nelec*madelung = -3.00286633348
DF object = <pyscf.pbc.dft.multigrid.MultiGridFFTDF object at 0x19f261b50>
XC functionals = pbe
small_rho_cutoff = 1e-07
Uniform grid, mesh = [113 113 113]
<class 'pyscf.pbc.dft.krks.KRKS'> does not have attributes  nlcgrids nlc
Set gradient conv threshold to 0.000316228
Traceback (most recent call last):
  File "/Users/janus/calculations/pyscf/pbc/test_multigrid.py", line 19, in <module>
    mf.kernel()
  File "<string>", line 2, in kernel
  File "/Users/janus/programs/pyscf/pyscf/scf/hf.py", line 1691, in scf
    kernel(self, self.conv_tol, self.conv_tol_grad,
  File "/Users/janus/programs/pyscf/pyscf/scf/hf.py", line 127, in kernel
    h1e = mf.get_hcore(mol)
  File "/Users/janus/programs/pyscf/pyscf/pbc/scf/khf.py", line 99, in get_hcore
    nuc = lib.asarray(mf.with_df.get_pp(kpts))
  File "/Users/janus/programs/pyscf/pyscf/pbc/dft/multigrid.py", line 385, in get_pp
    vpp = _get_j_pass2(mydf, vpplocG, hermi, kpts_lst)[0]
  File "/Users/janus/programs/pyscf/pyscf/pbc/dft/multigrid.py", line 856, in _get_j_pass2
    t_cell, coeff = t_cell.to_uncontracted_cartesian_basis()
  File "/Users/janus/programs/pyscf/pyscf/gto/mole.py", line 559, in to_uncontracted_cartesian_basis
    return decontract_basis(mol, to_cart=True)
  File "/Users/janus/programs/pyscf/pyscf/gto/mole.py", line 634, in decontract_basis
    lmax = mol._bas[ib0:ib1,ANG_OF].max()
  File "/Users/janus/anaconda3/envs/my-rdkit-env/lib/python3.9/site-packages/numpy/core/_methods.py", line 39, in _amax
    return umr_maximum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation maximum which has no identity
```

HEAD is at 6eb7542afc713 from yesterday. Thanks in advance.

diff --git a/pyscf/gto/mole.py b/pyscf/gto/mole.py
index 49f7fe9f34..636322d4c3 100644
--- a/pyscf/gto/mole.py
+++ b/pyscf/gto/mole.py
@@ -626,6 +626,9 @@ def _to_full_contraction(mol, bas_idx):

     aoslices = mol.aoslice_by_atom()
     for ia, (ib0, ib1) in enumerate(aoslices[:,:2]):
+        if ib0 == ib1: # No basis on atom ia
+            continue
+
         if atoms is not None:
             if isinstance(atoms, str):
                 to_apply = ((atoms == mol.atom_pure_symbol(ia)) or

diff --git a/pyscf/gto/test/test_mole.py b/pyscf/gto/test/test_mole.py
index 4adaa7e32a..1a91682db0 100644
--- a/pyscf/gto/test/test_mole.py
+++ b/pyscf/gto/test/test_mole.py
@@ -1033,6 +1033,11 @@ def test_decontract_basis(self):
         s = ctr_coeff.T.dot(pmol.intor('int1e_ovlp')).dot(ctr_coeff)
         self.assertAlmostEqual(abs(s - mol.intor('int1e_ovlp')).max(), 0, 12)

+        # discard basis on atom 2. (related to issue #1711)
+        mol._bas = mol._bas[:5]
+        pmol, c = mol.decontract_basis()
+        self.assertEqual(pmol.nbas, 14)
+
         mol = gto.M(atom='He',
                     basis=('ccpvdz', [[0, [5, 1]], [1, [3, 1]]]))
         pmol, contr_coeff = mol.decontract_basis()


Fix rsdf due to the basis sorting in 923d84c05 (issue #1942) (pyscf/pyscf#1947)

Possible bug in RSGDF

`RSGDF` fails (produces unreasonably high energies and doesn't converge) in a relatively simple case where `GDF` and `FFTDF` both provide reasonable results:

```
from pyscf.pbc import df, dft, gto
import numpy as np

def main():
    # unit cell
    a = np.array([[5.87704825, 0., 0.],
                  [-2.93852412, 5.08967308, 0.],
                  [0., 0., 9.4108361]])

    atom = '''
    Al   0.000000000000e+00   0.000000000000e+00   0.000000000000e+00;
    N    1.087472380182e-16   3.393115387675e+00   1.129307415351e+00;
    Al   1.087472380182e-16   3.393115387675e+00   4.705418050167e+00;
    N    0.000000000000e+00   0.000000000000e+00   5.834725465518e+00;
    '''
    cell = gto.M(a=a,
                 atom=atom,
                 unit='bohr',
                 basis='gth-szv',
                 pseudo={'Al': 'GTH-LDA', 'N': 'GTH-LDA'},
                 verbose=4,
                 precision=1.0e-12,
                 spin=0)
    cell.build()
    kpts = cell.make_kpts([1, 1, 1])

    # Regular GDF works fine for me (E = -23.5674653795703)
    #gdf = df.GDF(cell, kpts=kpts)

    # RSGDF doesn't work (unphysically high energies)
    gdf = df.RSGDF(cell, kpts=kpts)

    mf = dft.KRKS(cell, kpts=kpts, exxdiv=None, xc='PBE')
    mf.with_df = gdf
    mf.kernel()

if __name__ == '__main__':
    main()
```

diff --git a/pyscf/gto/mole.py b/pyscf/gto/mole.py
index 49f7fe9f34..43a47edaf4 100644
--- a/pyscf/gto/mole.py
+++ b/pyscf/gto/mole.py
@@ -1023,7 +1023,7 @@ def make_env(atoms, basis, pre_env=[], nucmod={}, nucprop={}):
     '''
     _atm = []
     _bas = []
-    _env = []
+    _env = [pre_env]
     ptr_env = len(pre_env)

     for ia, atom in enumerate(atoms):
@@ -1092,10 +1092,7 @@ def make_env(atoms, basis, pre_env=[], nucmod={}, nucprop={}):
         _bas = numpy.asarray(numpy.vstack(_bas), numpy.int32).reshape(-1, BAS_SLOTS)
     else:
         _bas = numpy.zeros((0,BAS_SLOTS), numpy.int32)
-    if _env:
-        _env = numpy.hstack((pre_env,numpy.hstack(_env)))
-    else:
-        _env = numpy.array(pre_env, copy=False)
+    _env = numpy.asarray(numpy.hstack(_env), dtype=numpy.float64)
     return _atm, _bas, _env

 def make_ecp_env(mol, _atm, ecp, pre_env=[]):
diff --git a/pyscf/pbc/df/rsdf_helper.py b/pyscf/pbc/df/rsdf_helper.py
index 719235e171..7fa0f74e2e 100644
--- a/pyscf/pbc/df/rsdf_helper.py
+++ b/pyscf/pbc/df/rsdf_helper.py
@@ -37,18 +37,21 @@
 """
 class MoleNoBasSort(mol_gto.mole.Mole):
     def build(self, **kwargs):
-        mol_gto.mole.Mole.build(self, **kwargs)
-
-        # sort self._bas
-        _bas = []
-        for iatm in range(self.natm):
-            atm = self.atom_symbol(iatm)
-            basin = self._basis[atm]
-            bas_ls = [b[0] for b in basin]
-            ls_uniq, ls_inv = np.unique(bas_ls, return_inverse=True)
-            order = np.argsort(np.concatenate([np.where(ls_inv==l)[0] for l in ls_uniq]))
-            _bas.append( self._bas[np.where(self._bas[:,0] == iatm)[0][order]] )
-        self._bas = np.vstack(_bas).astype(np.int32)
+        self.atom = kwargs.pop('atom')
+        self.basis = kwargs.pop('basis')
+        self._atom = mol_gto.format_atom(self.atom)
+        if isinstance(self.basis, dict):
+            self._basis = self.basis
+        else:
+            self._basis = {a[0]: self.basis for a in self._atom}
+
+        env = np.zeros(mol_gto.PTR_ENV_START)
+        # _bas should be constructed as it is in the input (see issue #1942).
+        self._atm, self._bas, self._env = self.make_env(
+            self._atom, self._basis, env)
+        self._built = True
+        return self
+
 def _remove_exp_basis_(bold, amin, amax):
     bnew = []
     for b in bold:

diff --git a/pyscf/pbc/df/test/test_rsdf.py b/pyscf/pbc/df/test/test_rsdf.py
index de00bcd0f3..d05a6633de 100644
--- a/pyscf/pbc/df/test/test_rsdf.py
+++ b/pyscf/pbc/df/test/test_rsdf.py
@@ -19,7 +19,7 @@
 from pyscf import ao2mo
 from pyscf.pbc import gto as pgto
 from pyscf.pbc import scf as pscf
-from pyscf.pbc.df import rsdf
+from pyscf.pbc.df import rsdf, df
 #from mpi4pyscf.pbc.df import df
 pyscf.pbc.DEBUG = False

@@ -124,6 +124,28 @@ def test_get_eri_0123(self):
         self.assertAlmostEqual(abs(eri0123.imag.sum()), 4.9887958509e-5, 7)
         self.assertAlmostEqual(lib.fp(eri0123), 0.9695261296288074-0.33222740818370966j, 8)

+    def test_rsdf_build(self):
+        cell = pgto.M(a=numpy.eye(3)*1.8,
+                      atom='''Li   0.   0.    0.; H    0.   .5   1.2 ''',
+                      basis={'Li': [[0, [5., 1.]], [0, [.6, 1.]], [1, [3., 1.]]],
+                             'H': [[0, [.3, 1.]]]})
+        auxbasis = {'Li': [[0, [5., 1.]], [0, [1.5, 1.]], [1, [.5, 1.]], [2, [2.5, 1.]]],
+                    'H':  [[0, [2., 1.]]]}
+        numpy.random.seed(2)
+        dm = numpy.random.random([cell.nao]*2)
+
+        gdf = df.GDF(cell)
+        gdf.auxbasis = auxbasis
+        jref, kref = gdf.get_jk(dm)
+
+        gdf = rsdf.RSGDF(cell)
+        gdf.auxbasis = auxbasis
+        vj, vk = gdf.get_jk(dm)
+
+        self.assertAlmostEqual(abs(vj - jref).max(), 0, 7)
+        self.assertAlmostEqual(abs(vk - kref).max(), 0, 7)
+        self.assertAlmostEqual(lib.fp(vj), 2.383648833459583, 7)
+        self.assertAlmostEqual(lib.fp(vk), 1.553598328349400, 7)

 if __name__ == '__main__':
     print("Full Tests for rsdf")


Fix PBC DFT Becke grid rounding error (pyscf/pyscf#1960)

PBC DFT Becke grid weights wrong

PySCF version: [fd8ee3e](https://github.com/pyscf/pyscf/commit/fd8ee3e264affb668874c543562a46239c66871e)
Input:
```
import numpy as np
from pyscf.pbc import gto, dft

atom = '''
Cu1    1.927800000000   1.927800000000   1.590250000000
Cu1    5.783400000000   5.783400000000   1.590250000000
Cu2    1.927800000000   5.783400000000   1.590250000000
Cu2    5.783400000000   1.927800000000   1.590250000000
O1     1.927800000000   3.855600000000   1.590250000000
O1     3.855600000000   5.783400000000   1.590250000000
O1     5.783400000000   3.855600000000   1.590250000000
O1     3.855600000000   1.927800000000   1.590250000000
O1     0.000000000000   1.927800000000   1.590250000000
O1     1.927800000000   7.711200000000   1.590250000000
O1     7.711200000000   5.783400000000   1.590250000000
O1     5.783400000000   0.000000000000   1.590250000000
Ca     0.000000000000   0.000000000000   0.000000000000
Ca     3.855600000000   3.855600000000   0.000000000000
Ca     7.711200000000   3.855600000000   0.000000000000
Ca     3.855600000000   7.711200000000   0.000000000000
'''

cell = gto.Cell()
cell.atom = atom
cell.a = np.array(
  [[7.7112,    0.,    0.],
   [0.,    7.7112,    0.],
   [0.,    0.,    3.1805]])
cell.verbose = 5
cell.build()

grids = dft.gen_grid.BeckeGrids(cell)
grids.build()
```
Output:
```
tot grids = 270432
cell vol = 1276.25053  sum(weights) = 1256.85607
```

diff --git a/pyscf/pbc/dft/gen_grid.py b/pyscf/pbc/dft/gen_grid.py
index d5b316a1f6..1cc4d9fa91 100644
--- a/pyscf/pbc/dft/gen_grid.py
+++ b/pyscf/pbc/dft/gen_grid.py
@@ -165,6 +165,7 @@ def get_becke_grids(cell, atom_grid={}, radi_method=dft.radi.gauss_chebyshev,
     b = cell.reciprocal_vectors(norm_to=1)
     supatm_idx = []
     k = 0
+    tol = 1e-15
     for iL, L in enumerate(Ls):
         for ia in range(cell.natm):
             coords, vol = atom_grids_tab[cell.atom_symbol(ia)]
@@ -174,24 +175,24 @@ def get_becke_grids(cell, atom_grid={}, radi_method=dft.radi.gauss_chebyshev,

             mask = np.ones(c.shape[1], dtype=bool)
             if dimension >= 1:
-                mask &= (c[0]>=-.5) & (c[0]<=.5)
+                mask &= (c[0]>-.5-tol) & (c[0]<.5+tol)
             if dimension >= 2:
-                mask &= (c[1]>=-.5) & (c[1]<=.5)
+                mask &= (c[1]>-.5-tol) & (c[1]<.5+tol)
             if dimension == 3:
-                mask &= (c[2]>=-.5) & (c[2]<=.5)
+                mask &= (c[2]>-.5-tol) & (c[2]<.5+tol)

             vol = vol[mask]
             if vol.size > 8:
                 c = c[:,mask]
                 if dimension >= 1:
-                    vol[c[0]==-.5] *= .5
-                    vol[c[0]== .5] *= .5
+                    vol[abs(c[0]+.5) < tol] *= .5
+                    vol[abs(c[0]-.5) < tol] *= .5
                 if dimension >= 2:
-                    vol[c[1]==-.5] *= .5
-                    vol[c[1]== .5] *= .5
+                    vol[abs(c[1]+.5) < tol] *= .5
+                    vol[abs(c[1]-.5) < tol] *= .5
                 if dimension == 3:
-                    vol[c[2]==-.5] *= .5
-                    vol[c[2]== .5] *= .5
+                    vol[abs(c[2]+.5) < tol] *= .5
+                    vol[abs(c[2]-.5) < tol] *= .5
                 coords = coords[mask]
                 coords_all.append(coords)
                 weights_all.append(vol)


Fix basis order issue in molden.load (issue #1961) (pyscf/pyscf#1963)

Re-ordering basis functions introduces mismatch

#1620 introduced ordering basis functions if they are not ordered correctly. When using `molden.load` to load molecules from molden files, the MO coefficients are not ordered accordingly, which leads to a mismatch and therefore wrong results.
One example would be the minimally augmented basis sets (like ma-def2-SVP) from ORCA, which have some s-type AOs appended after the p-type AOs from the standard def2-SVP.

The code has changed significantly since #1620, which makes fixing the problem a bit harder.

diff --git a/pyscf/gto/mole.py b/pyscf/gto/mole.py
index c8e460afef..b515875dee 100644
--- a/pyscf/gto/mole.py
+++ b/pyscf/gto/mole.py
@@ -419,7 +419,7 @@ def str2atm(line):
     return list(zip(z, c.tolist()))

 #TODO: sort exponents
-def format_basis(basis_tab):
+def format_basis(basis_tab, sort_basis=True):
     '''Convert the input :attr:`Mole.basis` to the internal data format.

     ``{ atom: [(l, ((-exp, c_1, c_2, ..),
@@ -457,10 +457,16 @@ def format_basis(basis_tab):
     fmt_basis = {}
     for atom, atom_basis in basis_tab.items():
         symb = _atom_symbol(atom)
-        fmt_basis[symb] = basis_converter(symb, atom_basis)
-
-        if len(fmt_basis[symb]) == 0:
+        _basis = basis_converter(symb, atom_basis)
+        if len(_basis) == 0:
             raise BasisNotFoundError('Basis not found for  %s' % symb)
+
+        # Sort basis accroding to angular momentum. This is important for method
+        # decontract_basis, which assumes that basis functions with the same
+        # angular momentum are grouped together. Related to issue #1620 #1770
+        if sort_basis:
+            _basis = sorted([b for b in _basis if b], key=lambda b: b[0])
+        fmt_basis[symb] = _basis
     return fmt_basis

 def _generate_basis_converter():
@@ -503,11 +509,6 @@ def converter(symb, raw_basis):
                     _basis.extend(nparray_to_list(rawb))
         else:
             _basis = nparray_to_list(raw_basis)
-
-        # Sort basis accroding to angular momentum. This is important for method
-        # decontract_basis, which assumes that basis functions with the same
-        # angular momentum are grouped together. Related to issue #1620 #1770
-        _basis = sorted([b for b in _basis if b], key=lambda b: b[0])
         return _basis
     return converter

@@ -654,6 +655,8 @@ def _to_full_contraction(mol, bas_idx):
             bas_idx = ib0 + numpy.where(mol._bas[ib0:ib1,ANG_OF] == l)[0]
             if len(bas_idx) == 0:
                 continue
+            if bas_idx[0] + len(bas_idx) != bas_idx[-1] + 1:
+                raise NotImplementedError('Discontinuous bases of same angular momentum')

             mol_exps, b_coeff = _to_full_contraction(mol, bas_idx)
             nprim, nc = b_coeff.shape
@@ -2523,11 +2526,13 @@ def build(self, dump_input=True, parse_arg=ARGPARSE,
         if self.verbose >= logger.WARN:
             self.check_sanity()

-        self._atom = self.format_atom(self.atom, unit=self.unit)
+        if self.atom:
+            self._atom = self.format_atom(self.atom, unit=self.unit)
         uniq_atoms = {a[0] for a in self._atom}

-        _basis = _parse_default_basis(self.basis, uniq_atoms)
-        self._basis = self.format_basis(_basis)
+        if self.basis:
+            _basis = _parse_default_basis(self.basis, uniq_atoms)
+            self._basis = self.format_basis(_basis)
         env = self._env[:PTR_ENV_START]
         self._atm, self._bas, self._env = \
                 self.make_env(self._atom, self._basis, env, self.nucmod,
@@ -2643,30 +2648,12 @@ def _build_symmetry(self, *args, **kwargs):
                            for ir in self.irrep_id]
         return self

-    @lib.with_doc(format_atom.__doc__)
-    def format_atom(self, atom, origin=0, axes=None, unit='Ang'):
-        return format_atom(atom, origin, axes, unit)
-
-    @lib.with_doc(format_basis.__doc__)
-    def format_basis(self, basis_tab):
-        return format_basis(basis_tab)
-
-    @lib.with_doc(format_pseudo.__doc__)
-    def format_pseudo(self, pseudo_tab):
-        return format_pseudo(pseudo_tab)
-
-    @lib.with_doc(format_ecp.__doc__)
-    def format_ecp(self, ecp_tab):
-        return format_ecp(ecp_tab)
-
-    @lib.with_doc(expand_etb.__doc__)
-    def expand_etb(self, l, n, alpha, beta):
-        return expand_etb(l, n, alpha, beta)
-
-    @lib.with_doc(expand_etbs.__doc__)
-    def expand_etbs(self, etbs):
-        return expand_etbs(etbs)
-    etbs = expand_etbs
+    format_atom = staticmethod(format_atom)
+    format_basis = staticmethod(format_basis)
+    format_pseudo = staticmethod(format_pseudo)
+    format_ecp = staticmethod(format_ecp)
+    expand_etb = staticmethod(expand_etb)
+    expand_etbs = etbs = staticmethod(expand_etbs)

     @lib.with_doc(make_env.__doc__)
     def make_env(self, atoms, basis, pre_env=[], nucmod={}, nucprop=None):
diff --git a/pyscf/tools/molden.py b/pyscf/tools/molden.py
index 8d0ac4e48f..07862101be 100644
--- a/pyscf/tools/molden.py
+++ b/pyscf/tools/molden.py
@@ -239,8 +239,16 @@ def read_one_bas(lsym, nb, fac=1):
         elif dat[0].upper() in 'SPDFGHIJ':
             basis[symb].append(read_one_bas(*dat))

-    mol.basis = envs['basis'] = basis
     mol.atom = [atoms[i] for i in atom_seq]
+    uniq_atoms = {a[0] for a in mol.atom}
+
+    # To avoid the mol.build() sort the basis, disable mol.basis and set the
+    # internal data _basis directly. It is a workaround to solve issue #1961.
+    # Mole.decontract_basis function should be rewritten to support
+    # discontinuous bases that have the same angular momentum.
+    mol.basis = {}
+    _basis = gto.mole._parse_default_basis(basis, uniq_atoms)
+    mol._basis = envs['basis'] = gto.format_basis(_basis, sort_basis=False)
     return mol

 def _parse_mo(lines, envs):
@@ -353,6 +361,8 @@ def load(moldenfile, verbose=0):
             else:
                 sys.stderr.write('Unknown section %s\n' % sec_title)

+    mo_energy, mo_coeff, mo_occ, irrep_labels, spins = None, None, None, None, None
+
     for sec_kind in _SEC_ORDER:
         if sec_kind == 'MO' and 'MO' in sec_kinds:
             if len(sec_kinds['MO']) == 1:
@@ -392,6 +402,8 @@ def load(moldenfile, verbose=0):
     if isinstance(mo_occ, tuple):
         mol.spin = int(mo_occ[0].sum() - mo_occ[1].sum())

+    if not mol._built:
+        mol.build(0, 0)
     return mol, mo_energy, mo_coeff, mo_occ, irrep_labels, spins

 parse = read = load
@@ -503,8 +515,9 @@ def remove_high_l(mol, mo_coeff=None):
     '''
     pmol = mol.copy()
     pmol.basis = {}
+    pmol._basis = {}
     for symb, bas in mol._basis.items():
-        pmol.basis[symb] = [b for b in bas if b[0] <= 4]
+        pmol._basis[symb] = [b for b in bas if b[0] <= 4]
     pmol.build(0, 0)
     if mo_coeff is None:
         return pmol, None


Chkfile initial guess problem in ROHF and ROKS (issue #1986) (pyscf/pyscf#2010)

geometry optimization with dispersion correction

Hi,
I tried to optimize a molecule with B3LYP-D3/D4 ([Simple DFT-D3](https://dftd3.readthedocs.io/en/latest/index.html)). But there are some errors.

```
from pyscf import gto, dft, scf
import dftd3.pyscf as disp
from pyscf.geomopt.geometric_solver import optimize

#no spin RHF
try:
    mol = gto.M(atom='O 0 0 0; O 0 0 1.8', unit='ANG', basis='631g*')
    mf = scf.RHF(mol)
    #mf.xc='b3lyp'
    grad=disp.energy(mf).nuc_grad_method().as_scanner()
    grad.optimizer().kernel()
    print("no spin RHF: OK")
except Exception as e:
    print(e)

#spin=1 hydroxides RHF
try:
    mol = gto.M(atom='H 0 0 0; O 0 0 1.8', unit='ANG', basis='631g*', spin=1)
    mf = scf.RHF(mol)
    #mf.xc='b3lyp'
    grad=disp.energy(mf).nuc_grad_method().as_scanner()
    grad.optimizer().kernel()
    print("spin=1 RHF: OK")
except Exception as e:
    print(e)

#spin=0 RKS
try:
    mol = gto.M(atom='O 0 0 0; O 0 0 1.8', unit='ANG', basis='631g*')
    mf = dft.RKS(mol)
    mf.xc='b3lyp'
    grad=disp.energy(mf).nuc_grad_method().as_scanner()
    grad.optimizer().kernel()
    print("spin=0 RKS: OK")
except Exception as e:
    print(e)

```
The above optimization works.

```
#spin=1 hydroxides RKS
try:
    mol = gto.M(atom='H 0 0 0; O 0 0 1.8', unit='ANG', basis='631g*', spin=1)
    mf = dft.RKS(mol)
    mf.xc='b3lyp'
    grad=disp.energy(mf).nuc_grad_method().as_scanner()
    grad.optimizer().kernel()
    print("spin=1 RKS: OK")
except Exception as e:
    print(e)

#spin=2  RKS
try:
    mol = gto.M(atom='O 0 0 0; O 0 0 1.8', unit='ANG', basis='631g*', spin=2)
    mf = dft.RKS(mol)
    mf.xc='b3lyp'
    grad=disp.energy(mf).nuc_grad_method().as_scanner()
    grad.optimizer().kernel()
    print("spin=2 RKS: OK")
except Exception as e:
    print(e)
```
When introducing spin to RKS, an error was raised: ''
mo_occ_a = (dm.mo_occ > 0).astype(numpy.double)
                ^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'list' and 'int'

When implementing UKS, it raised 'scf_grad must be an instance of Gradients'.
```
#spin=1 hydroxides UKS
try:
    mol = gto.M(atom='H 0 0 0; O 0 0 1.8', unit='ANG', basis='631g*', spin=1)
    mf = dft.UKS(mol)
    mf.xc='b3lyp'
    grad=disp.energy(mf).nuc_grad_method().as_scanner()
    grad.optimizer().kernel()
    print("spin UKS: OK")
except Exception as e:
    print(e)
```
So, it seems that 'spin' does not work for dft-d3 correction? And UKS is not supported?
I am not sure if these errors are rooted in pyscf or dftd3/d4 package.  But they all work well without dftd3.

diff --git a/.github/workflows/ci_linux/build_pyscf.sh b/.github/workflows/ci_linux/build_pyscf.sh
index 04e2aac97b..d9bd1b39e7 100755
--- a/.github/workflows/ci_linux/build_pyscf.sh
+++ b/.github/workflows/ci_linux/build_pyscf.sh
@@ -5,7 +5,7 @@ set -e
 cd ./pyscf/lib
 curl -L "https://github.com/pyscf/pyscf-build-deps/blob/master/pyscf-2.4a-deps.tar.gz?raw=true" | tar xzf -
 mkdir build; cd build
-cmake -DBUILD_LIBXC=OFF -DBUILD_XCFUN=OFF -DBUILD_LIBCINT=OFF -DXCFUN_MAX_ORDER=4 ..
+cmake -DBUILD_LIBXC=OFF -DBUILD_XCFUN=ON -DBUILD_LIBCINT=OFF -DXCFUN_MAX_ORDER=4 ..
 make -j4
 cd ..
 rm -Rf build
diff --git a/pyscf/scf/hf.py b/pyscf/scf/hf.py
index 8536176c47..37982c581f 100644
--- a/pyscf/scf/hf.py
+++ b/pyscf/scf/hf.py
@@ -656,12 +656,66 @@ def init_guess_by_chkfile(mol, chkfile_name, project=None):
     '''Read the HF results from checkpoint file, then project it to the
     basis defined by ``mol``

+    Kwargs:
+        project : None or bool
+            Whether to project chkfile's orbitals to the new basis.  Note when
+            the geometry of the chkfile and the given molecule are very
+            different, this projection can produce very poor initial guess.
+            In PES scanning, it is recommended to switch off project.
+
+            If project is set to None, the projection is only applied when the
+            basis sets of the chkfile's molecule are different to the basis
+            sets of the given molecule (regardless whether the geometry of
+            the two molecules are different).  Note the basis sets are
+            considered to be different if the two molecules are derived from
+            the same molecule with different ordering of atoms.
+
     Returns:
         Density matrix, 2D ndarray
     '''
-    from pyscf.scf import uhf
-    dm = uhf.init_guess_by_chkfile(mol, chkfile_name, project)
-    return dm[0] + dm[1]
+    from pyscf.scf import addons
+    chk_mol, scf_rec = chkfile.load_scf(chkfile_name)
+    if project is None:
+        project = not gto.same_basis_set(chk_mol, mol)
+
+    # Check whether the two molecules are similar
+    im1 = scipy.linalg.eigvalsh(mol.inertia_moment())
+    im2 = scipy.linalg.eigvalsh(chk_mol.inertia_moment())
+    # im1+1e-7 to avoid 'divide by zero' error
+    if abs((im1-im2)/(im1+1e-7)).max() > 0.01:
+        logger.warn(mol, "Large deviations found between the input "
+                    "molecule and the molecule from chkfile\n"
+                    "Initial guess density matrix may have large error.")
+
+    if project:
+        s = get_ovlp(mol)
+
+    def fproj(mo):
+        if project:
+            mo = addons.project_mo_nr2nr(chk_mol, mo, mol)
+            norm = numpy.einsum('pi,pi->i', mo.conj(), s.dot(mo))
+            mo /= numpy.sqrt(norm)
+        return mo
+
+    mo = scf_rec['mo_coeff']
+    mo_occ = scf_rec['mo_occ']
+    if getattr(mo[0], 'ndim', None) == 1:  # RHF
+        if numpy.iscomplexobj(mo):
+            raise NotImplementedError('TODO: project DHF orbital to UHF orbital')
+        mo_coeff = fproj(mo)
+        dm = make_rdm1(mo_coeff, mo_occ)
+    else:  #UHF
+        if getattr(mo[0][0], 'ndim', None) == 2:  # KUHF
+            logger.warn(mol, 'k-point UHF results are found.  Density matrix '
+                        'at Gamma point is used for the molecular SCF initial guess')
+            mo = mo[0]
+        dma = make_rdm1(fproj(mo[0]), mo_occ[0])
+        dmb = make_rdm1(fproj(mo[1]), mo_occ[1])
+        dm = dma + dmb
+        s = get_ovlp(mol)
+        _, mo_coeff = scipy.linalg.eigh(dm, s, type=2)
+        dm = lib.tag_array(dm, mo_coeff=mo_coeff[:,::-1], mo_occ=mo_occ)
+    return dm

 def get_init_guess(mol, key='minao'):
@@ -1358,11 +1412,9 @@ def __call__(self, mol_or_geom, **kwargs):
             dm0 = kwargs.pop('dm0')
         elif self.mo_coeff is None:
             dm0 = None
-        elif self.chkfile and h5py.is_hdf5(self.chkfile):
-            dm0 = self.from_chk(self.chkfile)
         else:
             dm0 = None
-            # dm0 form last calculation cannot be used in the current
+            # dm0 form last calculation may not be used in the current
             # calculation if a completely different system is given.
             # Obviously, the systems are very different if the number of
             # basis functions are different.
@@ -1371,6 +1423,8 @@ def __call__(self, mol_or_geom, **kwargs):
             # last calculation.
             if numpy.array_equal(self._last_mol_fp, mol.ao_loc):
                 dm0 = self.make_rdm1()
+            elif self.chkfile and h5py.is_hdf5(self.chkfile):
+                dm0 = self.from_chk(self.chkfile)
         self.mo_coeff = None  # To avoid last mo_coeff being used by SOSCF
         e_tot = self.kernel(dm0=dm0, **kwargs)
         self._last_mol_fp = mol.ao_loc
diff --git a/pyscf/scf/rohf.py b/pyscf/scf/rohf.py
index 777a9a5b17..f1243e9935 100644
--- a/pyscf/scf/rohf.py
+++ b/pyscf/scf/rohf.py
@@ -49,7 +49,28 @@ def init_guess_by_atom(mol):

 init_guess_by_huckel = uhf.init_guess_by_huckel
 init_guess_by_mod_huckel = uhf.init_guess_by_mod_huckel
-init_guess_by_chkfile = uhf.init_guess_by_chkfile
+
+def init_guess_by_chkfile(mol, chkfile_name, project=None):
+    '''Read SCF chkfile and make the density matrix for ROHF initial guess.
+
+    Kwargs:
+        project : None or bool
+            Whether to project chkfile's orbitals to the new basis.  Note when
+            the geometry of the chkfile and the given molecule are very
+            different, this projection can produce very poor initial guess.
+            In PES scanning, it is recommended to switch off project.
+
+            If project is set to None, the projection is only applied when the
+            basis sets of the chkfile's molecule are different to the basis
+            sets of the given molecule (regardless whether the geometry of
+            the two molecules are different).  Note the basis sets are
+            considered to be different if the two molecules are derived from
+            the same molecule with different ordering of atoms.
+    '''
+    dm = uhf.init_guess_by_chkfile(mol, chkfile_name, project)
+    mo_coeff = dm.mo_coeff[0]
+    mo_occ = dm.mo_occ[0] + dm.mo_occ[1]
+    return lib.tag_array(dm, mo_coeff=mo_coeff, mo_occ=mo_occ)

 def get_fock(mf, h1e=None, s1e=None, vhf=None, dm=None, cycle=-1, diis=None,
              diis_start_cycle=None, level_shift_factor=None, damp_factor=None):

diff --git a/pyscf/dft/test/test_he.py b/pyscf/dft/test/test_he.py
index 791593616d..27f6f0a08f 100644
--- a/pyscf/dft/test/test_he.py
+++ b/pyscf/dft/test/test_he.py
@@ -14,6 +14,7 @@
 # limitations under the License.

 import unittest
+import tempfile
 import numpy
 from pyscf import gto
 from pyscf import lib
@@ -191,6 +192,21 @@ def test_convert(self):
         self.assertTrue(isinstance(udks.to_dhf(), scf.dhf.DHF))
         self.assertTrue(isinstance(udks.to_dks('pbe'), dft.dks.DKS))

+    # issue 1986
+    def test_init_guess_chkfile(self):
+        with tempfile.NamedTemporaryFile() as tmpf:
+            mol = gto.M(atom='He 0 0 0', basis='631g', charge=1, spin=1)
+            mf = dft.RKS(mol)
+            mf.chkfile = tmpf.name
+            e1 = mf.kernel()
+            mf = dft.RKS(mol)
+            mf.init_guess = 'chkfile'
+            mf.chkfile = tmpf.name
+            mf.max_cycle = 1
+            e2 = mf.kernel()
+            self.assertAlmostEqual(e1, e2, 9)
+
+
 if __name__ == "__main__":
     print("Full Tests for He")
     unittest.main()
diff --git a/pyscf/dft/test/test_xc_deriv.py b/pyscf/dft/test/test_xc_deriv.py
index 03655f45e1..b6e430ffc2 100644
--- a/pyscf/dft/test/test_xc_deriv.py
+++ b/pyscf/dft/test/test_xc_deriv.py
@@ -415,7 +415,7 @@ def test_libxc_mgga_deriv3(self):
         self.assertAlmostEqual(xc1.sum(), 19.977512805950784, 7)
         self.assertAlmostEqual(abs(ref[1] - xc1).max(), 0, 9)

-    @unittest.skip(dft.libxc.max_deriv_order('pbe,') <= 3)
+    @unittest.skipIf(dft.libxc.max_deriv_order('pbe,') <= 3, 'libxc order')
     def test_libxc_gga_deriv4(self):
         rho1 = rho[:,:4].copy()
         xc1 = dft.libxc.eval_xc_eff('PBE', rho1, deriv=4)
@@ -425,7 +425,7 @@ def test_libxc_gga_deriv4(self):
         xc1 = dft.libxc.eval_xc_eff('PBE', rho1, deriv=4)
         self.assertAlmostEqual(xc1.sum(), -869.6617638095072, 4)

-    @unittest.skip(not hasattr(dft, 'xcfun'))
+    @unittest.skipIf(not hasattr(dft, 'xcfun'), 'xcfun order')
     def test_xcfun_lda_deriv3(self):
         rho1 = rho[:,0].copy()
         ref = eval_xc_eff('LDA,', rho1, 3, dft.xcfun)
@@ -457,7 +457,7 @@ def test_xcfun_lda_deriv3(self):
         self.assertAlmostEqual(xc1.sum(), -2.7477984980958627, 9)
         self.assertAlmostEqual(abs(ref[1] - xc1).max(), 0, 9)

-    @unittest.skip(not hasattr(dft, 'xcfun'))
+    @unittest.skipIf(not hasattr(dft, 'xcfun'), 'xcfun order')
     def test_xcfun_gga_deriv3(self):
         rho1 = rho[:,:4].copy()
         ref = eval_xc_eff('PBE', rho1, 3, dft.xcfun)
@@ -489,7 +489,7 @@ def test_xcfun_gga_deriv3(self):
         self.assertAlmostEqual(xc1.sum(), -3.0715856471099032, 9)
         self.assertAlmostEqual(abs(ref[1] - xc1).max(), 0, 9)

-    @unittest.skip(not hasattr(dft, 'xcfun'))
+    @unittest.skipIf(not hasattr(dft, 'xcfun'), 'xcfun order')
     def test_xcfun_mgga_deriv3(self):
         rho1 = rho
         ref = eval_xc_eff('M06', rho1, 3, dft.xcfun)
@@ -521,7 +521,7 @@ def test_xcfun_mgga_deriv3(self):
         self.assertAlmostEqual(xc1.sum(), 19.977512805950784, 9)
         self.assertAlmostEqual(abs(ref[1] - xc1).max(), 0, 9)

-    @unittest.skip(not (hasattr(dft, 'xcfun') and dft.xcfun.MAX_DERIV_ORDER > 3))
+    @unittest.skipIf(not (hasattr(dft, 'xcfun') and dft.xcfun.MAX_DERIV_ORDER > 3), 'xcfun order')
     def test_xcfun_gga_deriv4(self):
         rho1 = rho[:,:4].copy()
         xc1 = dft.xcfun.eval_xc_eff('PBE', rho1, deriv=4)


Improve cistring.addr2str for large address (issue #1886) (pyscf/pyscf#2050)

cistring error

Using `int32` for the indices causes issues when the range of indices exceeds the representable range of `int32`. For example, the following program crashes with `floating point exception (core dumped)`:
```python
from pyscf.fci import cistring

norb = 48
nelec = 24

string = 0b101100101000111111101011001001010001000100011011
address = cistring.str2addr(norb=norb, nelec=nelec, string=string)
string2 = cistring.addr2str(norb=norb, nelec=nelec, addr=address)
```
Additionally, these functions might succeed but give incorrect results. For example, this code does get to the assert statement, but that assertion fails:
```python
from pyscf.fci import cistring

string = 0b101101111101101111001110111111110111111100
address = cistring.str2addr(norb=63, nelec=32, string=string)
string2 = cistring.addr2str(norb=63, nelec=32, addr=address)
assert string == string2
```

diff --git a/pyscf/fci/cistring.py b/pyscf/fci/cistring.py
index 983178ff54..0ef463d693 100644
--- a/pyscf/fci/cistring.py
+++ b/pyscf/fci/cistring.py
@@ -343,12 +343,13 @@ def addr2str(norb, nelec, addr):
     '''Convert CI determinant address to string'''
     if norb >= 64:
         raise NotImplementedError('norb >= 64')
-    assert num_strings(norb, nelec) > addr
+    max_addr = num_strings(norb, nelec)
+    assert max_addr > addr

-    if addr < 2**31:
+    if max_addr < 2**31:
         return addrs2str(norb, nelec, [addr])[0]
-
-    return _addr2str(norb, nelec, addr)
+    else:
+        return _addr2str(norb, nelec, addr)

 def _addr2str(norb, nelec, addr):
     if addr == 0 or nelec == norb or nelec == 0:

diff --git a/pyscf/fci/test/test_cistring.py b/pyscf/fci/test/test_cistring.py
index 634d92a7cd..dcd5d6f988 100644
--- a/pyscf/fci/test/test_cistring.py
+++ b/pyscf/fci/test/test_cistring.py
@@ -156,6 +156,12 @@ def test_addr2str(self):
         self.assertEqual(bin(cistring.addr2str(6, 3, 8)), '0b11010')
         self.assertEqual(bin(cistring.addr2str(7, 4, 9)), '0b110011')

+        # Test large addresss
+        string = 0b101101111101101111001110111111110111111100
+        address = cistring.str2addr(norb=63, nelec=32, string=string)
+        string2 = cistring.addr2str(norb=63, nelec=32, addr=address)
+        self.assertEqual(string, string2)
+
     def test_str2addr(self):
         self.assertEqual(str2addr(6, 3, int('0b11001' ,2)), cistring.str2addr(6, 3, int('0b11001' ,2)))
         self.assertEqual(str2addr(6, 3, int('0b11010' ,2)), cistring.str2addr(6, 3, int('0b11010' ,2)))


Check cylindrical symmetry of the orbitals for FCI solver (issue #2022) (pyscf/pyscf#2062)

CASSCF with symmetry for B-H throws an error

Hi,

Calculating the CAS(2,2) energy of B-H with `symmetry=True` throws an error. For Li-H or for `symmetry=False` or for CAS(3,2) there is no error. Issue #1548 seemed to be related. Also, I am running PySCF 2.4.0.

Could someone help figure this out? Thanks!

```
from pyscf import gto, scf, mcscf

### CAS(3,2) runs fine
mol1 = gto.M(atom="B 0 0 0; H 1.5 0 0", basis="sto3g", symmetry=True)
mc1 = mcscf.CASSCF(scf.RHF(mol1).run(), 3, 2)
mc1.kernel()

### Li-H runs fine
mol2 = gto.M(atom="Li 0 0 0; H 1.5 0 0", basis="sto3g", symmetry=True)
mc2 = mcscf.CASSCF(scf.RHF(mol2).run(), 2, 2)
mc2.kernel()

### symmetry=False runs fine
mol3 = gto.M(atom="B 0 0 0; H 1.5 0 0", basis="sto3g", symmetry=False)
mc3 = mcscf.CASSCF(scf.RHF(mol3).run(), 2, 2)
mc3.kernel()

### throws error
mol4 = gto.M(atom="B 0 0 0; H 1.5 0 0", basis="sto3g", symmetry=True)
mc4 = mcscf.CASSCF(scf.RHF(mol4).run(), 2, 2)
mc4.kernel()
```

the output is

```
converged SCF energy = -24.7216853309577
/Users/vrn738/PycharmProjects/mrpt2/venv/lib/python3.11/site-packages/numpy/linalg/linalg.py:2180: RuntimeWarning: divide by zero encountered in det
  r = _umath_linalg.det(a, signature=signature)
/Users/vrn738/PycharmProjects/mrpt2/venv/lib/python3.11/site-packages/numpy/linalg/linalg.py:2180: RuntimeWarning: invalid value encountered in det
  r = _umath_linalg.det(a, signature=signature)
CASSCF energy = -24.7615485480989
CASCI E = -24.7615485480989  E(CI) = -1.32807642898465  S^2 = 0.0000000
converged SCF energy = -7.86335762153512
CASSCF energy = -7.88099257585260
CASCI E = -7.88099257585260  E(CI) = -1.09902483008292  S^2 = 0.0000000
converged SCF energy = -24.7216853309577
CASSCF energy = -24.7435717749394
CASCI E = -24.7435717749394  E(CI) = -1.29911474076125  S^2 = 0.0000000
converged SCF energy = -24.7216853309577
Traceback (most recent call last):
  File "/Users/vrn738/PycharmProjects/mrpt2/molecules/BH/casscf_symmetry_issue.py", line 20, in <module>
    mc4.kernel()
  File "/Users/vrn738/PycharmProjects/mrpt2/venv/lib/python3.11/site-packages/pyscf/mcscf/mc1step_symm.py", line 73, in kernel
    _kern(self, mo_coeff,
  File "/Users/vrn738/PycharmProjects/mrpt2/venv/lib/python3.11/site-packages/pyscf/mcscf/mc1step.py", line 351, in kernel
    e_tot, e_cas, fcivec = casscf.casci(mo, ci0, eris, log, locals())
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vrn738/PycharmProjects/mrpt2/venv/lib/python3.11/site-packages/pyscf/mcscf/mc1step.py", line 878, in casci
    e_tot, e_cas, fcivec = casci.kernel(fcasci, mo_coeff, ci0, log,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vrn738/PycharmProjects/mrpt2/venv/lib/python3.11/site-packages/pyscf/mcscf/casci.py", line 608, in kernel
    e_tot, fcivec = casci.fcisolver.kernel(h1eff, eri_cas, ncas, nelecas,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vrn738/PycharmProjects/mrpt2/venv/lib/python3.11/site-packages/pyscf/fci/direct_spin1_symm.py", line 781, in kernel
    ci0 = self.get_init_guess(norb, nelec, nroots, hdiag, orbsym, wfnsym_ir)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vrn738/PycharmProjects/mrpt2/venv/lib/python3.11/site-packages/pyscf/fci/direct_spin1_symm.py", line 724, in get_init_guess
    ci0 = get_init_guess_cyl_sym(
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vrn738/PycharmProjects/mrpt2/venv/lib/python3.11/site-packages/pyscf/fci/direct_spin1_symm.py", line 291, in get_init_guess_cyl_sym
    ca = _cyl_sym_csf2civec(strsa, addra, orbsym, degen_mapping)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vrn738/PycharmProjects/mrpt2/venv/lib/python3.11/site-packages/pyscf/fci/direct_spin1_symm.py", line 348, in _cyl_sym_csf2civec
    u = _cyl_sym_orbital_rotation(orbsym, degen_mapping)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vrn738/PycharmProjects/mrpt2/venv/lib/python3.11/site-packages/pyscf/fci/direct_spin1_symm.py", line 369, in _cyl_sym_orbital_rotation
    u[j,j] = sqrthi
    ~^^^^^
IndexError: index 2 is out of bounds for axis 0 with size 2
```

diff --git a/pyscf/fci/direct_spin1_cyl_sym.py b/pyscf/fci/direct_spin1_cyl_sym.py
index 0056bbd992..80aa27192f 100644
--- a/pyscf/fci/direct_spin1_cyl_sym.py
+++ b/pyscf/fci/direct_spin1_cyl_sym.py
@@ -42,9 +42,9 @@
 from pyscf.fci import cistring
 from pyscf.fci import direct_spin1
 from pyscf.fci import direct_spin1_symm
-from pyscf.fci.direct_spin1_symm import (_sv_associated_det,
-                                         _strs_angular_momentum,
-                                         _cyl_sym_orbital_rotation)
+from pyscf.fci.direct_spin1_symm import (
+    _sv_associated_det, _strs_angular_momentum, _cyl_sym_orbital_rotation,
+    _validate_degen_mapping)
 from pyscf.fci import direct_nosym
 from pyscf.fci import addons
 from pyscf import __config__
@@ -558,6 +558,10 @@ def kernel(self, h1e, eri, norb, nelec, ci0=None,
         if not hasattr(orbsym, 'degen_mapping'):
             degen_mapping = map_degeneracy(h1e.diagonal(), orbsym)
             orbsym = lib.tag_array(orbsym, degen_mapping=degen_mapping)
+        if not _validate_degen_mapping(orbsym.degen_mapping, norb):
+            raise lib.exceptions.PointGroupSymmetryError(
+                'Incomplete 2D-irrep orbitals for cylindrical symmetry.\n'
+                f'orbsym = {orbsym}.')

         u = _cyl_sym_orbital_rotation(orbsym, orbsym.degen_mapping)
         h1e = u.dot(h1e).dot(u.conj().T)
diff --git a/pyscf/fci/direct_spin1_symm.py b/pyscf/fci/direct_spin1_symm.py
index 272a432ec6..017505836a 100644
--- a/pyscf/fci/direct_spin1_symm.py
+++ b/pyscf/fci/direct_spin1_symm.py
@@ -257,6 +257,13 @@ def get_init_guess(norb, nelec, nroots, hdiag, orbsym, wfnsym=0):
         raise RuntimeError(f'Initial guess for symmetry {wfnsym} not found')
     return ci0

+def _validate_degen_mapping(mapping, norb):
+    '''Check if 2D irreps are properly paired'''
+    mapping = np.asarray(mapping)
+    return (mapping.max() < norb and
+            # Must be self-conjugated
+            numpy.array_equal(mapping[mapping], numpy.arange(norb)))
+
 def get_init_guess_cyl_sym(norb, nelec, nroots, hdiag, orbsym, wfnsym=0):
     neleca, nelecb = _unpack_nelec(nelec)
     strsa = strsb = cistring.gen_strings4orblist(range(norb), neleca)
@@ -751,6 +758,11 @@ def kernel(self, h1e, eri, norb, nelec, ci0=None,
                 orbsym = lib.tag_array(orbsym, degen_mapping=degen_mapping)
             if davidson_only is None:
                 davidson_only = True
+            if not _validate_degen_mapping(orbsym.degen_mapping, norb):
+                raise lib.exceptions.PointGroupSymmetryError(
+                    'Incomplete 2D-irrep orbitals for cylindrical symmetry.\n'
+                    f'orbsym = {orbsym}. '
+                    f'Retry {self.__class__} with D2h subgroup symmetry.')

         wfnsym_ir = self.guess_wfnsym(norb, nelec, ci0, orbsym, wfnsym, **kwargs)
         self.sym_allowed_idx = sym_allowed_indices(nelec, orbsym, wfnsym_ir)

diff --git a/pyscf/fci/test/test_spin1_cyl_sym.py b/pyscf/fci/test/test_spin1_cyl_sym.py
index 84e014d690..0648528e31 100644
--- a/pyscf/fci/test/test_spin1_cyl_sym.py
+++ b/pyscf/fci/test/test_spin1_cyl_sym.py
@@ -212,6 +212,15 @@ def test_linearmole_a2(self):
         mc.run()
         self.assertAlmostEqual(mc.e_tot, 2.8999951068356475, 8)

+    def test_incomplete_orbsym(self):
+        sol = direct_spin1_cyl_sym.FCI(gto.Mole())
+        no, ne = 2, 2
+        h1 = np.ones((no,no))
+        h2 = np.ones((no,no,no,no))
+        orbsym = lib.tag_array(np.array([0,3]), degen_mapping=[0,2])
+        with self.assertRaises(lib.exceptions.PointGroupSymmetryError):
+            sol.kernel(h1, h2, no, ne, orbsym=orbsym)
+
 if __name__ == "__main__":
     print("Full Tests for spin1-symm")
     unittest.main()
diff --git a/pyscf/fci/test/test_spin1_symm.py b/pyscf/fci/test/test_spin1_symm.py
index 462009e3dd..4b63d48d27 100644
--- a/pyscf/fci/test/test_spin1_symm.py
+++ b/pyscf/fci/test/test_spin1_symm.py
@@ -15,7 +15,7 @@

 import unittest
 import numpy
-from pyscf import gto
+from pyscf import gto, lib
 from pyscf import scf
 from pyscf import ao2mo
 from pyscf import fci
@@ -197,6 +197,17 @@ def test_linearmole(self):
         ci1 = fci.addons.transform_ci(ci_y, (3,3), u.T)
         self.assertAlmostEqual(abs(ci1.ravel().dot(ci_y.ravel())), 1, 9)

+    def test_incomplete_orbsym(self):
+        mol = gto.Mole()
+        mol.groupname = 'Dooh'
+        sol = direct_spin1_symm.FCI(mol)
+        no, ne = 2, 2
+        h1 = numpy.ones((no,no))
+        h2 = numpy.ones((no,no,no,no))
+        orbsym = lib.tag_array(numpy.array([0,3]), degen_mapping=[0,2])
+        with self.assertRaises(lib.exceptions.PointGroupSymmetryError):
+            sol.kernel(h1, h2, no, ne, orbsym=orbsym)
+
 if __name__ == "__main__":
     print("Full Tests for spin1-symm")
     unittest.main()


Improve to_hf for pbc dft (pyscf/pyscf#2172)

* Fix to_hf method for k-point dft methods
* FIx old compiler compatibility issue

KeyError in pyscf.pbc.df.df_jk?

With
```
from pyscf.pbc import gto, scf

cell = gto.Cell()
cell.atom='''
C 0.000000000000   0.000000000000   0.000000000000
C 1.685068664391   1.685068664391   1.685068664391
'''
cell.basis = 'gth-szv'
cell.pseudo = 'gth-pade'
cell.a = '''
0.000000000, 3.370137329, 3.370137329
3.370137329, 0.000000000, 3.370137329
3.370137329, 3.370137329, 0.000000000'''
cell.unit = 'B'
cell.verbose = 5
cell.build()

#
# KDFT and KGW with 2x2x2 k-points
#
kpts = cell.make_kpts([2,2,2])
kmf = scf.KRKS(cell, xc='pbe').density_fit()
kmf.kpts = kpts
emf = kmf.kernel()

myrhf = kmf.to_hf()
veff = myrhf.get_veff()

```

I get the following error:

```
Traceback (most recent call last):
  File "/home/chillenb/src/gwscripts/kpttest.py", line 28, in <module>
    veff = myrhf.get_veff()
           ^^^^^^^^^^^^^^^^
  File "/home/chillenb/mambaforge/envs/rttddft/lib/python3.11/site-packages/pyscf/pbc/scf/khf.py", line 541, in get_veff
    vj, vk = self.get_jk(cell, dm_kpts, hermi, kpts, kpts_band)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chillenb/mambaforge/envs/rttddft/lib/python3.11/site-packages/pyscf/pbc/scf/khf.py", line 529, in get_jk
    vj, vk = self.with_df.get_jk(dm_kpts, hermi, kpts, kpts_band,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chillenb/mambaforge/envs/rttddft/lib/python3.11/site-packages/pyscf/pbc/df/df.py", line 443, in get_jk
    vk = df_jk.get_k_kpts(self, dm, hermi, kpts, kpts_band, exxdiv)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chillenb/mambaforge/envs/rttddft/lib/python3.11/site-packages/pyscf/pbc/df/df_jk.py", line 613, in get_k_kpts
    make_kpt(ki, kj, True)
  File "/home/chillenb/mambaforge/envs/rttddft/lib/python3.11/site-packages/pyscf/pbc/df/df_jk.py", line 446, in make_kpt
    for LpqR, LpqI, sign in mydf.sr_loop((kpti,kptj), max_memory, False):
  File "/home/chillenb/mambaforge/envs/rttddft/lib/python3.11/site-packages/pyscf/pbc/df/df.py", line 355, in sr_loop
    with _load3c(self._cderi, self._dataname, kpti_kptj) as j3c:
  File "/home/chillenb/mambaforge/envs/rttddft/lib/python3.11/site-packages/pyscf/pbc/df/df.py", line 692, in __enter__
    return self.getitem(*self.kpti_kptj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chillenb/mambaforge/envs/rttddft/lib/python3.11/site-packages/pyscf/pbc/df/df.py", line 746, in getitem
    raise KeyError(f'Key {key} not found')
KeyError: 'Key j3c/8 not found'
```

The fact that the 3c2e integrals should now be computed, and `_j_only` reset, is not getting detected. Is there a recommended way to do this manually?

Would it be bad design to add an option to scf.KRKS.density_fit() so that, if I need to calculate Vk later, I can override the default value of _j_only?

(PySCF 2.5.0 installed from conda-forge)

diff --git a/pyscf/lib/dft/utils.c b/pyscf/lib/dft/utils.c
index 04ef8e5b2f..fc27b4a48f 100644
--- a/pyscf/lib/dft/utils.c
+++ b/pyscf/lib/dft/utils.c
@@ -52,11 +52,20 @@ void get_gga_vrho_gs(double complex *out, double complex *vrho_gs, double comple
     int i;
     int ngrid2 = 2 * ngrid;
     double complex fac = -2. * _Complex_I;
-    #pragma omp parallel for simd schedule(static)
+#pragma omp parallel
+{
+    double complex v;
+// ensure OpenMP 4.0
+#if defined _OPENMP && _OPENMP >= 201307
+    #pragma omp for simd schedule(static)
+#else
+    #pragma omp for schedule(static)
+#endif
     for (i = 0; i < ngrid; i++) {
-        out[i] = ( Gv[i*3]   * vsigma1_gs[i]
-                  +Gv[i*3+1] * vsigma1_gs[i+ngrid]
-                  +Gv[i*3+2] * vsigma1_gs[i+ngrid2]) * fac + vrho_gs[i];
-        out[i] *= weight;
+        v = ( Gv[i*3]   * vsigma1_gs[i]
+             +Gv[i*3+1] * vsigma1_gs[i+ngrid]
+             +Gv[i*3+2] * vsigma1_gs[i+ngrid2]) * fac + vrho_gs[i];
+        out[i] = v * weight;
     }
 }
+}
diff --git a/pyscf/pbc/dft/kgks.py b/pyscf/pbc/dft/kgks.py
index 7774f2e36e..fb9e1f2f68 100644
--- a/pyscf/pbc/dft/kgks.py
+++ b/pyscf/pbc/dft/kgks.py
@@ -146,7 +146,14 @@ def nuc_grad_method(self):

     def to_hf(self):
         '''Convert to KGHF object.'''
-        from pyscf.pbc import scf
-        return self._transfer_attrs_(scf.KGHF(self.cell, self.kpts))
+        from pyscf.pbc import scf, df
+        out = self._transfer_attrs_(scf.KGHF(self.cell, self.kpts))
+
+        # Pure functionals only construct J-type integrals. Enable all integrals for KHF.
+        if (not self._numint.libxc.is_hybrid_xc(self.xc) and
+            len(self.kpts) > 1 and getattr(self.with_df, '_j_only', False)):
+            out.with_df._j_only = False
+            out.with_df.reset()
+        return out

     to_gpu = lib.to_gpu
diff --git a/pyscf/pbc/dft/krks.py b/pyscf/pbc/dft/krks.py
index 6278605de9..d690647879 100644
--- a/pyscf/pbc/dft/krks.py
+++ b/pyscf/pbc/dft/krks.py
@@ -181,25 +181,13 @@ def nuc_grad_method(self):

     def to_hf(self):
         '''Convert to KRHF object.'''
-        from pyscf.pbc import scf
-        return self._transfer_attrs_(scf.KRHF(self.cell, self.kpts))
+        from pyscf.pbc import scf, df
+        out = self._transfer_attrs_(scf.KRHF(self.cell, self.kpts))
+        # Pure functionals only construct J-type integrals. Enable all integrals for KHF.
+        if (not self._numint.libxc.is_hybrid_xc(self.xc) and
+            len(self.kpts) > 1 and getattr(self.with_df, '_j_only', False)):
+            out.with_df._j_only = False
+            out.with_df.reset()
+        return out

     to_gpu = lib.to_gpu
-
-
-if __name__ == '__main__':
-    from pyscf.pbc import gto
-    cell = gto.Cell()
-    cell.unit = 'A'
-    cell.atom = 'C 0.,  0.,  0.; C 0.8917,  0.8917,  0.8917'
-    cell.a = '''0.      1.7834  1.7834
-                1.7834  0.      1.7834
-                1.7834  1.7834  0.    '''
-
-    cell.basis = 'gth-szv'
-    cell.pseudo = 'gth-pade'
-    cell.verbose = 7
-    cell.output = '/dev/null'
-    cell.build()
-    mf = KRKS(cell, cell.make_kpts([2,1,1]))
-    print(mf.kernel())
diff --git a/pyscf/pbc/dft/kroks.py b/pyscf/pbc/dft/kroks.py
index 1d2a1198f8..640bf05e97 100644
--- a/pyscf/pbc/dft/kroks.py
+++ b/pyscf/pbc/dft/kroks.py
@@ -61,25 +61,13 @@ def dump_flags(self, verbose=None):

     def to_hf(self):
         '''Convert to KROHF object.'''
-        from pyscf.pbc import scf
-        return self._transfer_attrs_(scf.KROHF(self.cell, self.kpts))
+        from pyscf.pbc import scf, df
+        out = self._transfer_attrs_(scf.KROHF(self.cell, self.kpts))
+        # Pure functionals only construct J-type integrals. Enable all integrals for KHF.
+        if (not self._numint.libxc.is_hybrid_xc(self.xc) and
+            len(self.kpts) > 1 and getattr(self.with_df, '_j_only', False)):
+            out.with_df._j_only = False
+            out.with_df.reset()
+        return out

     to_gpu = lib.to_gpu
-
-
-if __name__ == '__main__':
-    from pyscf.pbc import gto
-    cell = gto.Cell()
-    cell.unit = 'A'
-    cell.atom = 'C 0.,  0.,  0.; C 0.8917,  0.8917,  0.8917'
-    cell.a = '''0.      1.7834  1.7834
-                1.7834  0.      1.7834
-                1.7834  1.7834  0.    '''
-
-    cell.basis = 'gth-szv'
-    cell.pseudo = 'gth-pade'
-    cell.verbose = 7
-    cell.output = '/dev/null'
-    cell.build()
-    mf = KROKS(cell, cell.make_kpts([2,1,1]))
-    print(mf.kernel())
diff --git a/pyscf/pbc/dft/kuks.py b/pyscf/pbc/dft/kuks.py
index ac66f973bc..732adcfa32 100644
--- a/pyscf/pbc/dft/kuks.py
+++ b/pyscf/pbc/dft/kuks.py
@@ -157,25 +157,13 @@ def nuc_grad_method(self):

     def to_hf(self):
         '''Convert to KUHF object.'''
-        from pyscf.pbc import scf
-        return self._transfer_attrs_(scf.KUHF(self.cell, self.kpts))
+        from pyscf.pbc import scf, df
+        out = self._transfer_attrs_(scf.KUHF(self.cell, self.kpts))
+        # Pure functionals only construct J-type integrals. Enable all integrals for KHF.
+        if (not self._numint.libxc.is_hybrid_xc(self.xc) and
+            len(self.kpts) > 1 and getattr(self.with_df, '_j_only', False)):
+            out.with_df._j_only = False
+            out.with_df.reset()
+        return out

     to_gpu = lib.to_gpu
-
-
-if __name__ == '__main__':
-    from pyscf.pbc import gto
-    cell = gto.Cell()
-    cell.unit = 'A'
-    cell.atom = 'C 0.,  0.,  0.; C 0.8917,  0.8917,  0.8917'
-    cell.a = '''0.      1.7834  1.7834
-                1.7834  0.      1.7834
-                1.7834  1.7834  0.    '''
-
-    cell.basis = 'gth-szv'
-    cell.pseudo = 'gth-pade'
-    cell.verbose = 7
-    cell.output = '/dev/null'
-    cell.build()
-    mf = KUKS(cell, cell.make_kpts([2,1,1]))
-    print(mf.kernel())

diff --git a/pyscf/pbc/dft/test/test_kgks.py b/pyscf/pbc/dft/test/test_kgks.py
index a4fba351c4..6ebd1165ea 100644
--- a/pyscf/pbc/dft/test/test_kgks.py
+++ b/pyscf/pbc/dft/test/test_kgks.py
@@ -22,6 +22,7 @@
 from pyscf import lib
 from pyscf.pbc import gto as gto
 from pyscf.pbc import dft as dft
+from pyscf.pbc import scf as pbcscf
 from pyscf.pbc.df import rsdf_builder, gdf_builder
 try:
     import mcfun
@@ -189,6 +190,19 @@ def test_mcol_x2c_kgks_lda(self):
         mf.run()
         self.assertAlmostEqual(mf.e_tot, -1.4910121442258883, 6)

+    def test_to_hf(self):
+        mf = dft.KGKS(cell).density_fit()
+        mf.with_df._j_only = True
+        a_hf = mf.to_hf()
+        self.assertTrue(a_hf.with_df._j_only)
+        self.assertTrue(isinstance(a_hf, pbcscf.kghf.KGHF))
+
+        mf = dft.KGKS(cell, kpts=cell.make_kpts([2,1,1])).density_fit()
+        mf.with_df._j_only = True
+        a_hf = mf.to_hf()
+        self.assertTrue(not a_hf.with_df._j_only)
+        self.assertTrue(isinstance(a_hf, pbcscf.kghf.KGHF))
+

 if __name__ == '__main__':
     print("Full Tests for pbc.dft.kgks")
diff --git a/pyscf/pbc/dft/test/test_krks.py b/pyscf/pbc/dft/test/test_krks.py
index 090711ec54..e8418cb972 100644
--- a/pyscf/pbc/dft/test/test_krks.py
+++ b/pyscf/pbc/dft/test/test_krks.py
@@ -22,6 +22,7 @@

 from pyscf.pbc import gto as pbcgto
 from pyscf.pbc import dft as pbcdft
+from pyscf.pbc import scf as pbcscf

 def build_cell(mesh):
@@ -143,6 +144,31 @@ def test_rsh_df(self):
         mf.kernel()
         self.assertAlmostEqual(mf.e_tot, -2.4766238116030683, 5)

+    def test_to_hf(self):
+        mf = pbcdft.KRKS(cell).density_fit()
+        mf.with_df._j_only = True
+        a_hf = mf.to_hf()
+        self.assertTrue(a_hf.with_df._j_only)
+        self.assertTrue(isinstance(a_hf, pbcscf.khf.KRHF))
+
+        mf = pbcdft.KRKS(cell, kpts=cell.make_kpts([2,1,1])).density_fit()
+        mf.with_df._j_only = True
+        a_hf = mf.to_hf()
+        self.assertTrue(not a_hf.with_df._j_only)
+        self.assertTrue(isinstance(a_hf, pbcscf.khf.KRHF))
+
+        mf = pbcdft.KROKS(cell).density_fit()
+        mf.with_df._j_only = True
+        a_hf = mf.to_hf()
+        self.assertTrue(a_hf.with_df._j_only)
+        self.assertTrue(isinstance(a_hf, pbcscf.krohf.KROHF))
+
+        mf = pbcdft.KROKS(cell, kpts=cell.make_kpts([2,1,1])).density_fit()
+        mf.with_df._j_only = True
+        a_hf = mf.to_hf()
+        self.assertTrue(not a_hf.with_df._j_only)
+        self.assertTrue(isinstance(a_hf, pbcscf.krohf.KROHF))
+
 # TODO: test the reset method of pbcdft.KRKS, pbcdft.RKS whether the reset
 # methods of all subsequent objects are called

diff --git a/pyscf/pbc/dft/test/test_kuks.py b/pyscf/pbc/dft/test/test_kuks.py
index d99fddf7fa..a81ae3051f 100644
--- a/pyscf/pbc/dft/test/test_kuks.py
+++ b/pyscf/pbc/dft/test/test_kuks.py
@@ -19,6 +19,7 @@
 import unittest
 import numpy as np
 from pyscf.pbc import gto as pbcgto
+from pyscf.pbc import scf as pbcscf
 from pyscf.pbc import dft as pbcdft

@@ -91,6 +92,19 @@ def test_rsh_df(self):
         mf.kernel()
         self.assertAlmostEqual(mf.e_tot, -2.4766238116030683, 7)

+    def test_to_hf(self):
+        mf = pbcdft.KUKS(cell).density_fit()
+        mf.with_df._j_only = True
+        a_hf = mf.to_hf()
+        self.assertTrue(a_hf.with_df._j_only)
+        self.assertTrue(isinstance(a_hf, pbcscf.kuhf.KUHF))
+
+        mf = pbcdft.KUKS(cell, kpts=cell.make_kpts([2,1,1])).density_fit()
+        mf.with_df._j_only = True
+        a_hf = mf.to_hf()
+        self.assertTrue(not a_hf.with_df._j_only)
+        self.assertTrue(isinstance(a_hf, pbcscf.kuhf.KUHF))
+

 if __name__ == '__main__':
     print("Full Tests for pbc.dft.kuks")


Remove the ase module bulk from pyscf_ase (fix #2271) (pyscf/pyscf#2279)

Fix pyscf_ase (fix #2271)
Fix the TDDFT get_ab method for RSH functions (fix #2261)

Issue calculating static polarizability using RSH exchange-correlation functionals

Hello, everyone. I need to have access and freedom to work with the A and B matrices from coupled-perturbed KS. So, I'm trying to use the following code together with the CAM-B3LYP functional for static polarizability. However, the code doesn't work for this functional or any other RSH tested. It is curious since the code works properly for any other XC functionals, e.g., GGA, Hybrids, etc. I see someone with a similar issue and the fix was to change from LibXC to XCfun. But, this is not working for me. Can someone help me, please?

Results using the attached code:
Static polarizability from reference: 1.4907486930182852 (the right answer)
Static polarizability is: 1.3051766863495733

PS1: I also tried "a, b = tddft.TDDFT(mf).get_ab()". I still have the same issue with RSHs.

PS2: Using "rks.Polarizability(mf).polarizability()" works with RSH functionals (that's how I debug the results, i.e., reference). However, I don't know how to or if I have access to A and B using this code.

Thanks.

[cpks.zip](https://github.com/user-attachments/files/15873707/cpks.zip)

Cannot import name 'bulk' from 'ase.lattice'

I just tried to call the member function `from_ase()` with `ase 3.23.0` and `pyscf 2.6.1` and got an error. I guess `ase` has changed the lattice module. Simply removing the import works, as the function `bulk` is not used anywhere in `pyscf_ase.py`.

https://github.com/pyscf/pyscf/blob/d4ce357503bc865d43a956c8f540f8c0f38ef5cb/pyscf/pbc/tools/pyscf_ase.py#L27

diff --git a/pyscf/pbc/tools/pyscf_ase.py b/pyscf/pbc/tools/pyscf_ase.py
index f2d24a14fa..7c27c0654a 100644
--- a/pyscf/pbc/tools/pyscf_ase.py
+++ b/pyscf/pbc/tools/pyscf_ase.py
@@ -24,7 +24,6 @@
 import numpy as np
 from ase.calculators.calculator import Calculator
 import ase.dft.kpoints
-from ase.lattice import bulk

 def pyscf_to_ase_atoms(cell):
     '''
diff --git a/pyscf/tdscf/dhf.py b/pyscf/tdscf/dhf.py
index 5e3eb0f27c..0cf824353f 100644
--- a/pyscf/tdscf/dhf.py
+++ b/pyscf/tdscf/dhf.py
@@ -127,6 +127,7 @@ def add_hf_(a, b, hyb=1):
             raise NotImplementedError('DKS-TDDFT for NLC functionals')

         omega, alpha, hyb = ni.rsh_and_hybrid_coeff(mf.xc, mol.spin)
+        assert omega == 0.

         a, b = add_hf_(a, b, hyb)

diff --git a/pyscf/tdscf/ghf.py b/pyscf/tdscf/ghf.py
index dc23b63e0c..822fe128f7 100644
--- a/pyscf/tdscf/ghf.py
+++ b/pyscf/tdscf/ghf.py
@@ -162,6 +162,7 @@ def add_hf_(a, b, hyb=1):
             raise NotImplementedError

         omega, alpha, hyb = ni.rsh_and_hybrid_coeff(mf.xc, mol.spin)
+        assert omega == 0.

         a, b = add_hf_(a, b, hyb)

diff --git a/pyscf/tdscf/rhf.py b/pyscf/tdscf/rhf.py
index 720c3dade4..655e7026b9 100644
--- a/pyscf/tdscf/rhf.py
+++ b/pyscf/tdscf/rhf.py
@@ -152,6 +152,13 @@ def add_hf_(a, b, hyb=1):
         omega, alpha, hyb = ni.rsh_and_hybrid_coeff(mf.xc, mol.spin)

         add_hf_(a, b, hyb)
+        if omega != 0:  # For RSH
+            with mol.with_range_coulomb(omega):
+                eri_mo = ao2mo.general(mol, [orbo,mo,mo,mo], compact=False)
+                eri_mo = eri_mo.reshape(nocc,nmo,nmo,nmo)
+                k_fac = alpha - hyb
+                a -= numpy.einsum('ijba->iajb', eri_mo[:nocc,:nocc,nocc:,nocc:]) * k_fac
+                b -= numpy.einsum('jaib->iajb', eri_mo[:nocc,nocc:,:nocc,nocc:]) * k_fac

         xctype = ni._xc_type(mf.xc)
         dm0 = mf.make_rdm1(mo_coeff, mo_occ)
diff --git a/pyscf/tdscf/uhf.py b/pyscf/tdscf/uhf.py
index 1812926c6f..2692b9b688 100644
--- a/pyscf/tdscf/uhf.py
+++ b/pyscf/tdscf/uhf.py
@@ -188,6 +188,19 @@ def add_hf_(a, b, hyb=1):
         omega, alpha, hyb = ni.rsh_and_hybrid_coeff(mf.xc, mol.spin)

         add_hf_(a, b, hyb)
+        if omega != 0:  # For RSH
+            with mol.with_range_coulomb(omega):
+                eri_aa = ao2mo.general(mol, [orbo_a,mo_a,mo_a,mo_a], compact=False)
+                eri_bb = ao2mo.general(mol, [orbo_b,mo_b,mo_b,mo_b], compact=False)
+                eri_aa = eri_aa.reshape(nocc_a,nmo_a,nmo_a,nmo_a)
+                eri_bb = eri_bb.reshape(nocc_b,nmo_b,nmo_b,nmo_b)
+                a_aa, a_ab, a_bb = a
+                b_aa, b_ab, b_bb = b
+                k_fac = alpha - hyb
+                a_aa -= numpy.einsum('ijba->iajb', eri_aa[:nocc_a,:nocc_a,nocc_a:,nocc_a:]) * k_fac
+                b_aa -= numpy.einsum('jaib->iajb', eri_aa[:nocc_a,nocc_a:,:nocc_a,nocc_a:]) * k_fac
+                a_bb -= numpy.einsum('ijba->iajb', eri_bb[:nocc_b,:nocc_b,nocc_b:,nocc_b:]) * k_fac
+                b_bb -= numpy.einsum('jaib->iajb', eri_bb[:nocc_b,nocc_b:,:nocc_b,nocc_b:]) * k_fac

         xctype = ni._xc_type(mf.xc)
         dm0 = mf.make_rdm1(mo_coeff, mo_occ)

diff --git a/pyscf/tdscf/test/test_tdrks.py b/pyscf/tdscf/test/test_tdrks.py
index 0df6b2f8c1..2ca7cbb540 100644
--- a/pyscf/tdscf/test/test_tdrks.py
+++ b/pyscf/tdscf/test/test_tdrks.py
@@ -107,6 +107,15 @@ def test_tddft_b3lyp(self):
         ref = diagonalize(a, b, nroots=5) * 27.2114
         self.assertAlmostEqual(abs(es - ref).max(), 0, 7)

+    def test_tddft_camb3lyp(self):
+        mf = mol.RKS(xc='camb3lyp').run()
+        td = mf.TDDFT()
+        es = td.kernel(nstates=4)[0]
+        a,b = td.get_ab()
+        e_ref = diagonalize(a, b, 5)
+        self.assertAlmostEqual(abs(es[:3]-e_ref[:3]).max(), 0, 8)
+        self.assertAlmostEqual(lib.fp(es[:3]*27.2114), 9.0054057603534, 4)
+
     def test_tda_b3lypg(self):
         mf = dft.RKS(mol)
         mf.xc = 'b3lypg'
diff --git a/pyscf/tdscf/test/test_tduks.py b/pyscf/tdscf/test/test_tduks.py
index 9679fc5ea5..61a70172ff 100644
--- a/pyscf/tdscf/test/test_tduks.py
+++ b/pyscf/tdscf/test/test_tduks.py
@@ -130,6 +130,15 @@ def test_tddft_b3lyp(self):
         es = td.kernel(nstates=4)[0] * 27.2114
         self.assertAlmostEqual(lib.fp(es[:3]), 1.2984822994759448, 4)

+    def test_tddft_camb3lyp(self):
+        mf = mol1.UKS(xc='camb3lyp').run()
+        td = mf.TDDFT()
+        es = td.kernel(nstates=4)[0]
+        a,b = td.get_ab()
+        e_ref = diagonalize(a, b, 5)
+        self.assertAlmostEqual(abs(es[:3]-e_ref[:3]).max(), 0, 8)
+        self.assertAlmostEqual(lib.fp(es[:3]*27.2114), 7.69383202636, 4)
+
     def test_tda_b3lyp(self):
         td = tdscf.TDA(mf_b3lyp).set(conv_tol=1e-12)
         es = td.kernel(nstates=4)[0] * 27.2114


Fix addons.smearing RHF electron number (pyscf/pyscf#2290)

Smearing electron number bug

Smeared RHF with odd nelec gives incorrect electron number.

```
from pyscf import gto, scf
from pyscf.scf import addons

mol = gto.Mole()
mol.atom = '''
    7      0.   0  -0.7
    7      0.   0   0.7'''
mol.basis = 'cc-pvdz'
mol.charge = -1
mol.spin = 1
mol.build()
mf = scf.RHF(mol)
mf = addons.smearing(mf, sigma=0.1)
mf.kernel()
print(mf.mol.nelectron)
print(mf.mo_occ.sum())
```

```
converged SCF energy = -42.2391465363651
15
16.0
```

diff --git a/pyscf/scf/addons.py b/pyscf/scf/addons.py
index a120087f33..4f26a5367b 100644
--- a/pyscf/scf/addons.py
+++ b/pyscf/scf/addons.py
@@ -83,7 +83,7 @@ def nelec_cost_fn(m, mo_es, nocc):

 def _get_fermi(mo_energy, nocc):
     mo_e_sorted = numpy.sort(mo_energy)
-    return mo_e_sorted[nocc-1]
+    return mo_e_sorted[numpy.ceil(nocc).astype(int) - 1]

 class _SmearingSCF:

@@ -174,7 +174,7 @@ def get_occ(self, mo_energy=None, mo_coeff=None):
             else:
                 mo_es = mo_energy
             if is_rhf:
-                nocc = (nelectron + 1) // 2
+                nocc = nelectron / 2

             if self.mu0 is None:
                 mu, mo_occs = _smearing_optimize(f_occ, mo_es, nocc, sigma)

diff --git a/pyscf/scf/test/test_addons.py b/pyscf/scf/test/test_addons.py
index 88595191cd..76ed2a5cba 100644
--- a/pyscf/scf/test/test_addons.py
+++ b/pyscf/scf/test/test_addons.py
@@ -459,6 +459,27 @@ def test_rohf_smearing(self):
         self.assertAlmostEqual(myhf_s.e_tot, -243.086989253, 5)
         self.assertAlmostEqual(myhf_s.entropy, 17.11431, 4)

+    def test_rhf_smearing_nelec(self):
+        mol = gto.Mole()
+        mol.verbose = 5
+        mol.output = '/dev/null'
+        mol.atom = '''
+            7      0.   0  -0.7
+            7      0.   0   0.7'''
+        mol.basis = 'cc-pvdz'
+        mol.charge = +1
+        mol.spin = 1
+        mol.build()
+        mf = scf.RHF(mol)
+        mf = addons.frac_occ(mf)
+        e_frac = mf.kernel()
+
+        mf_smear = scf.RHF(mol)
+        mf_smear = addons.smearing(mf_smear, sigma=1e-5, method='fermi')
+        e_smear = mf_smear.kernel()
+        self.assertAlmostEqual(mf.mo_occ.sum(), mf_smear.mo_occ.sum(), 5)
+        self.assertAlmostEqual(e_frac, e_smear, 9)
+
     def test_smearing_mu0(self):
         def _hubbard_hamilts_pbc(L, U):
             h1e = numpy.zeros((L, L))


Fix bugs in complex-valued contractions for fci.direct_nosym (fix #2302) (pyscf/pyscf#2305)

Additionally, update grad.sacasscf due to API changes in scipy-1.14

contract_2e gives inconsistent results on complex vector

The following code generates a two-body tensor and a random complex-valued FCI vector, and then contracts the two-body tensor with the vector in two ways:
- Directly using `contract_2e`
- Using `contract_2e` on the real and imaginary parts of the vector and adding the results

These should give the same result, but they don't.

```python
import math

import numpy as np
import pyscf
import pyscf.mcscf
from pyscf.fci.direct_nosym import absorb_h1e, contract_2e

# Generate a two-body tensor
mol = pyscf.gto.Mole()
mol.build(
    atom=[["H", (0, 0, 0)], ["H", (1.0, 0, 0)]],
    basis="sto-6g",
    symmetry="Dooh",
    verbose=0,
)
scf = pyscf.scf.RHF(mol).run()
n_electrons = int(sum(scf.mo_occ))
n_alpha = (n_electrons + mol.spin) // 2
n_beta = (n_electrons - mol.spin) // 2
nelec = (n_alpha, n_beta)
norb = mol.nao_nr()
cas = pyscf.mcscf.CASCI(scf, norb, nelec)
h1e, _ = cas.get_h1cas()
h2e = cas.get_h2cas()
eri = absorb_h1e(h1e, h2e, norb, nelec, 0.5)

# Generate a random state vector
rng = np.random.default_rng(1234)
dim = math.comb(norb, n_alpha) * math.comb(norb, n_beta)
vec = rng.standard_normal(dim) + 1j * rng.standard_normal(dim)
vec /= np.linalg.norm(vec)

# Contract directly
result = contract_2e(eri, vec, norb, nelec)
# Contract real and imaginary parts separately
result_alt = contract_2e(eri, vec.real, norb, nelec) + 1j * contract_2e(
    eri, vec.imag, norb, nelec
)

print(np.allclose(result, result_alt))
print(np.linalg.norm(result - result_alt))
```
```
False
1.0875338674395465
```

diff --git a/pyscf/fci/direct_nosym.py b/pyscf/fci/direct_nosym.py
index fcd67f5664..913cedd6b0 100644
--- a/pyscf/fci/direct_nosym.py
+++ b/pyscf/fci/direct_nosym.py
@@ -125,7 +125,7 @@ def contract_2e(eri, fcivec, norb, nelec, link_index=None):
     outI += contract_2e(eriI, ciR, norb, nelec, link_index=link_index)
     out = outR.astype(numpy.complex128)
     out.imag = outI
-    return outR
+    return out

 def absorb_h1e(h1e, eri, norb, nelec, fac=1):
     '''Modify 2e Hamiltonian to include 1e Hamiltonian contribution.
diff --git a/pyscf/grad/lagrange.py b/pyscf/grad/lagrange.py
index b93707865e..89ff6d99a0 100644
--- a/pyscf/grad/lagrange.py
+++ b/pyscf/grad/lagrange.py
@@ -14,6 +14,7 @@
 # limitations under the License.

 import numpy as np
+import scipy
 from scipy import linalg, optimize
 from scipy.sparse import linalg as sparse_linalg
 from pyscf import lib, __config__
@@ -126,9 +127,9 @@ def my_Lvec_last ():
         prec_obj = sparse_linalg.LinearOperator ((self.nlag,self.nlag), matvec=precond,
                                                  dtype=bvec.dtype)
         x0_guess = self.get_init_guess (bvec, Adiag, Aop, precond)
-        Lvec, info_int = sparse_linalg.cg(Aop_obj, -bvec, x0=x0_guess,
-                                          atol=self.conv_atol,
-                                          maxiter=self.max_cycle, callback=my_call, M=prec_obj)
+        Lvec, info_int = _cg(Aop_obj, -bvec, x0=x0_guess,
+                             tol=self.conv_rtol, atol=self.conv_atol,
+                             maxiter=self.max_cycle, callback=my_call, M=prec_obj)
         logger.info (self, ('Lagrange multiplier determination {} after {} iterations\n'
                             '   |geff| = {}, |Lvec| = {}').format (
                                 'converged' if info_int == 0 else 'not converged',
@@ -190,3 +191,10 @@ def __call__(self, x):
         Adiagd[abs(Adiagd)<1e-8] = 1e-8
         x /= Adiagd
         return x
+
+if scipy.__version__.split('.')[1] >= '14': # scipy 1.14
+    def _cg(A, b, x0=None, *, tol=1e-05, atol=0.0, maxiter=None, M=None, callback=None):
+        return sparse_linalg.cg(A, b, x0, rtol=tol, atol=atol, maxiter=maxiter,
+                                M=M, callback=callback)
+else:
+    _cg = sparse_linalg.cg

diff --git a/pyscf/fci/test/test_direct_nosym.py b/pyscf/fci/test/test_direct_nosym.py
index d3beef460c..cda504ded4 100644
--- a/pyscf/fci/test/test_direct_nosym.py
+++ b/pyscf/fci/test/test_direct_nosym.py
@@ -60,6 +60,9 @@ def test_contract_complex(self):
         ci1 += 1j * fci.direct_nosym.contract_2e(h2e, ci0.imag, norb, nelec)
         self.assertTrue(numpy.allclose(ci1ref, ci1))

+        ci1 = fci.direct_nosym.contract_2e(h2e, ci0, norb, nelec).astype(complex)
+        self.assertAlmostEqual(abs(ci1ref - ci1).max(), 0, 12)
+
     def test_absorb_h1e(self):
         href = fci_slow.absorb_h1e(h1e, h2e, norb, nelec)
         h1 = fci.direct_nosym.absorb_h1e(h1e, h2e, norb, nelec)