# Triplet FFLO Superconductivity in the doped Kitaev-Heisenberg Honeycomb Model

## Abstract

We provide analytical and numerical evidence of a spin-triplet FFLO superconductivity in the itinerant Kitaev-Heisenberg model (anti-ferromagnetic Kitaev coupling and ferromagnetic Heisenberg coupling) on the honeycomb lattice around quarter filling. The strong spin-orbit coupling in our model leads to the emergence of 6 inversion symmetry centers for the Fermi surface at non zero momenta in the first Brillouin zone. We show how the Cooper pairs condense into these non-trivial momenta, causing the spatial modulation of the superconducting order parameter. Applying a Ginzburg-Landau expansion analysis, we find that the superconductivity has three separated degenerate ground states with three different spin-triplet pairings. This picture is also supported by exact diagonalizations on finite clusters.

Introduction- Mott insulator and high- superconductor are closely related since the latter can be obtained from doping the half-filled Mott insulator Anderson (); F.C.Zhang (); Sigrist (); Anderson_Lee (); Karyn_Maurice (). One key element in superconductivity is the emergence of off-diagonal long-range order which results in the Bardeen-Cooper-Schrieffer ground state where Cooper pairs have a zero net momentum. The pairing, proposed by C. N. Yang Yang (), binds electrons with momenta and , and therefore involves a superconductivity with non-zero Cooper pair momentum. This superconductivity is referred to as the Fulde-Ferrell-Larkin-Ovchinnikov (FFLO) superconductivity Fulde_Ferrell (); Larkin (); Larkin1 (). The FFLO superconductivity, which supports a spatial modulation for the electron pairing due to the non-trivial Cooper pair momentum, was first proposed in the ’60s in a system with significant Zeeman interaction, which shifts the Fermi surfaces for the up and down spins. Experimental realizations of FFLO superconductivity have been proposed, for example, in heavy-fermions Sarrao (), ultra-cold atom systems Liao (); MartinWolfgang (); Zi_Cai (); Mora (); Chevy_Mora (); Koponen (), BEC analogues Ivana () and in magnetic analogue materials Green (). However, this exotic phase of matter has not yet been conclusively observed, especially in real materials.

Lately, the studies of “iridates”, a family of materials with significant spin-orbit coupling, have aroused great interests KrempaKim (); Shitade (); Rau () partly because of the emergence of topological Mott physics pesin () and its connection to the Kitaev anyon model Kitaev (); Halasz (). It has been shown both theoretically and experimentally that the existence of zigzag-magnetic order results from a Kitaev-Heisenberg magnetic coupling in the two-dimensional sodium iridate family Chaloupka (); Jackeli_Khaliullin (); Y.Singh (); Y.Singh1 (); Reuther (). An additional symmetric-off diagonal exchange term can also be added in the analysis Chun (). Doping these spin-orbit Mott insulators has been addressed theoretically Rosenow (); Ashvin () and has started to attract some experimental attention Cao (). Here, we address superconductivity in the presence of a large Hubbard interaction and adopt a localized magnetism point of view where the Kitaev-Heisenberg spin Hamiltonian originates from super-exchange processes Hassan_senechal (). Such Kitaev-Heisenberg physics can also be realized in cold atom systems Duan (). Using both analytical and numerical methods, we provide convincing evidences of a spin-triplet FFLO superconductor thanks to the spin-orbit coupling close to quarter-filling without breaking the time-reversal symmetry. Superconductivity around quarter-filling has also attracted some attention in the context of graphene Nandkishore (); Annica ().

Before showing detailed derivations, we summarize the main points. The Kitaev-Heisenberg coupling entails spin-triplet pairing that engenders spinor-condensates leggett (); Zibold (); Corre () in momentum space. One important ingredient here is the appearance of 6 inversion symmetry centers for the Fermi surface at non zero momenta in the first Brillouin zone. This will allow the Cooper pairs with triplet pairing to condense at non-trivial momenta. In Fig. 1, we show the band structure of the spin-orbit coupling model and the symmetry centers of the Fermi surface. Electron pairs around these symmetry centers with non-trivial momenta form spin-triplet pairs with Cooper pair momenta . We shall study the superconductivity by calculating the Cooper pairs’ response in the Ginzburg-Landau theory for both spin-triplet and spin-singlet pairing. We provide compelling evidence of a triplet FFLO superconductor through a Ginzburg-Landau expansion and an exact diagonalization analysis.

Model Hamiltonian- For the doped Kitaev-Heisenberg model, we consider the following Hamiltonian on the honeycomb lattice:

(1) |

here and refer to the site index, and to electron operators on the lattices A and B in Fig. 1a. and are the spins of the electrons and the Pauli matrix with respectively for red, green and blue links () and take other components than (See Fig. 1a). We note the Gutzwiller projectors as or according to the sub-lattice Edegger (); Rice_Ueda (); Rice_Ueda2 (). The filling factor and the doping level are connected by the relation: . In contrast to previous analyses Ashvin (); Rosenow (), we include a spin-orbit term of the (doped) model Hassan_senechal (), such that the anti-ferromagnetic Kitaev and ferromagnetic Heisenberg couplings at half-filling are microscopically obtained from second-order super-exchange processes: with U the Hubbard interaction. Setting and , we recover the model used in Ref. Chaloupka () describing the half-filled system. One shall assume that is real to avoid an induced Dzyaloshinskii-Moriya interaction. However, an imaginary does not change the physics in the limit of . With a purely imaginary , the time-reversal symmetry (TRS) is restored and we will show the presence of FFLO superconductivity with TRS in this limit.

Band structure around quarter-filling- Around quarter-filling, which is sufficiently away from half-filling, one can assume that the effect of the Gutzwiller weights on the values of is weak and neglect the renormalization of . We can then diagonalize :

(2) |

in which and , and (). We see that in the spin-orbit coupling limit () the Fermi surface has six additional inversion symmetry centers, apart from the inversion symmetry center with trivial momentum , in the first Brillouin zone (FBZ) () as indicated in Fig. 1b. This derives from the Sine function remaining invariant under the change of . In Fig. 1c, we show the band structure at the spin-orbit coupling limit : the four bands have a conic structure for the Fermi surface at half and quarter filling.

Superconducting Instability- The doped itinerant Kitaev-Heisenberg model in the spin-orbit limit () has symmetry centers around quarter-filling with momenta: () and . There are 4 kinds of Cooper pairs around these symmetry centers Center symmetry (); supplement ():

(3) |

In the direct space, the three types of spin-triplet pairing and the spin-singlet pairing in competition are:

(4) |

The Kitaev-Heisenberg coupling involves the density channel besides the superconductivity pairing. We have checked that around quarter-filling the density channel renormalizes the spin-orbit coupling term and such renormalization is negligible supplement (). Then we can decompose the Kitaev-Heisenberg coupling at the mean-field level as:

(5) |

in which is the Fourier transform of the order parameter in Eq. 4 with spatial phase modulation . denotes here the number of unit cells.

We constitute the Nambu spinor for the four Cooper pairs ( is defined in Eq. 2) and write down their Gor’kov-Green function (, FBZ). We then pursue the Landau expansion supplement (). In the spin-orbit coupling limit (, ), we have the second order Landau expansion (here we fix following Ref. Rosenow ()):

(6) |

in which is the free energy and to the lowest (second) order is proportional to the inverse of the Cooper pair vertex function supplement (). When , we have checked that is negligible because of frustration in the momentum space; therefore we focus our attention on the diagonal part of the inverse of the Cooper pair vertex function that we denote as . When , the triplet superconductor pairing is stable AltlandSimons (). In Fig. 3, we show as a function of FBZ at temperature , in which we remark the condensation of spin-triplet Cooper pairs into the peaks at wave vector . We have three spin-triplet condensates at different momenta as shown in Fig. 2 a, b and c.

We also study the peak of the static Cooper pair response as a function of temperature at different doping levels : the peak remains finite at quarter-filling, while it has logarithmic divergence at zero temperature when the doping diverts from quarter-filling (Fig. 4a). Here, is proportional to the density of states at the Fermi level, which vanishes linearly as , which means that at quarter-filling superconductivity disappears and we have a free electron system, assuming is not too large compared to . At low temperature, the peak of the condensate profile stays positive while the peak of the spin-singlet condensate profile remains negative at all temperature (Fig. 4b). This indicates that in the spin-orbit coupling limit, the doped itinerant Kitaev-Heisenberg model hosts only the three spin-triplet ground states.

The three spin-triplet condensates may interact with each other and we have calculated the box diagram in Fig. 5 to study this effect by extending the Landau expansion to the fourth order. In Fig. 5, we note

(7) |

as the creation operators for the three Cooper pairs. Since the three Cooper pairs condense at different momenta , the box diagram is actually the only one respecting momentum conservation. To the fourth order, we obtain the free energy of the three condensates:

(8) | |||||

in which and are positive numbers obtained from the calculation of the box diagram in Fig. 5. We have checked that and thus we deduce that mixing of the three superconducting condensates is not energetically favorable, and there is phase separation among the three types of fermionic pairs. Consequentially, the ground state wave function at zero temperature is three times degenerate (See Fig. 2): the modulated (Eq. 4) are represented by bold and dashed lines ((a) red for X, (b) green for Y and (c) blue for Z).

When and are small compared to and , the three FFLO states are still stable when the temperature is low enough ( . The FFLO phase remains stable as long as the energy related to the critical temperature is bigger than the gap of the free electron system around quarter-filling opened by the term i.e. .

Exact Diagonalization of the Kitaev-Heisenberg model- We have done an exact diagonalization of the Kitaev-Heisenberg model of Eq. 1 in the spin-orbit coupling limit . The exact diagonalization treats the Gutzwiller projectors exactly in Eq. 1. We fix the parametrization , (here we choose as suggested by Ref. Rosenow ()). The system has plaquettes with periodic boundary conditions in both directions, and is filled with electrons on the sites. and are both even numbers in order to avoid frustration of the FFLO condensates. Due to computational constraints, we reduce our study to three system sizes: , . For an odd number of Cooper pairs (doped system), the lowest energy eigenstates appear in momentum sectors and (in the bases of , .) as shown in Figs. 6b, d, and g. The degeneracy for the three spin-triplet states is partially lifted when which breaks the symmetry of a rotation followed by a permutation of spin components. For an even number of Cooper pairs, the ground state appears in momentum sector as shown in Figs. 6a, c, e, f and h. In agreement with the theory, coincides with the three discrete version of the FFLO Cooper pair momenta () for an odd number of Cooper pairs while for even number of Cooper pairs . This alternation of ground state momentum sector as a function of particle numbers distinguishes the FFLO superconductivity here from other modulated orders like spin or charge density waves supplement (). The quasi-degeneracy in Fig.6b is yet to be understood and might just be a finite size effect.

Conclusion- We have provided both analytical and numerical evidence of a pure spin-triplet FFLO superconductor in the doped itinerant Kitaev-Heisenberg model in the spin-orbit coupling limit (). When is purely imaginary, the time-reversal symmetry (TRS) is restored. The key ingredient of the FFLO superconductivity here is the symmetry centers of the Fermi surface at non-trivial momenta instead of a Zeeman field. The ground state is three times degenerate with respectively the three spin-triplet pairing in the p-wave state with non-trivial Cooper pair momentum and spatial moduation of phase in the direction of lattice vector for the order parameter. These results may have relevance for doped iridate honeycomb materials or in ultra-cold atom systems.

Acknowledgments- We acknowledge discussions with Sylvain Capponi, Claudio Castelnovo, Fabrice Gerbier, Loic Herviou, Dmitry Kovrizhin, Claudine Lacroix, Philippe Lecheminant, Frédéric Mila, Christophe Mora, Catherine Pépin, Alexandru Petrescu, Didier Poilblanc, and Julien Vidal. N.R. was supported by the Princeton Global Scholarship. K.L.H. has benefited from discussions at KITP Santa-Barbara and CIFAR meetings in Canada, and was supported in part by the National Science Foundation under Grant No. PHY11-25915.

## Appendix A Supplemental material

In the supplementary material, we provide additional analytical and numerical results that might be relevant to a more specialized audience.

### a.1 Spin-Triplet Pairings

In the weak coupling limit (free electrons), we can measure the spin of the 4 electron pairs as shown in Fig. 1a at the Fermi level using the projector as ( for the first and fourth band for the second and third band). The functions are given in Eq. 2. In the limit of if we measure the spin of the two electron pairs at the Fermi level connected by that we denote as and , we have:

(9) |

Similarly for the spin expectation values of the electron pairs connected by and have the following relation:

(10) |

In particular, for the electron pairs with zero Cooper pair momentum, we have:

(11) |

We can see obviously from Eq. 9 and Eq. 10 that electron pairs around symmetry center minimize the coupling in the form of Kitaev-Heisenberg and the electron pairs around the inversion center minimize the coupling in the form of Heisenberg . In the language of wave function of the electron pairs, we can interpret the above as a tensor of two spin wave functions with respectively electron momentum and . We denote here as spin up in the polarization (). For example, for electron pair wave function with Cooper pair momentum , we will have:

(12) |

In very much the same way, we have wave functions for electron pairs with Cooper pair momentum and :

(13) |

If we express the above wave function (Eq. 13) with electron creation operators, we will obtain the spin-triplet pairing in Eq. 3:

(14) |

We have checked numerically that there exists uniform nearest-neighbor hopping amplitudes induced by the Kitaev-Heisenberg coupling on the link (). This spin-dependent amplitudes renormalizes the spin-orbit coupling, however such renormalization is negligible () which justifies the mean-field decomposition in Eq. 5.

### a.2 Gor’kov-Green function

Here, we detail the construction of the Gor’kov-Green function and specific form of the Cooper pair vertex function given in Eq. 6. We can first write down the Green function for the free electron with the wave function :

(15) |

in which is given in Eq. 2 and , are respectively the energy dispersion and band projectors for the I-th band. The Gor’kov-Green function for the superconductor is obtained from the free electron Green function and the non-diagonal terms denote the pairing of the electrons.

Taking into account Eq. 5, we can write down Gor’kov-Green function, with the Nambu spinor , which couples free electron wave function with momentum and :

(16) |

and and are matrices describing the superconductivity pairing in the basis of and are the Pauli matrices in the spin subspace.

(17) |

With Eq. 6, one can write down the lowest (second) order correction to the free energy with the vertex function which denotes the electron-electron or the hole-hole interaction in the Cooper pair channel:

(18) |

in which and are the indices and energies for the bands, and the corresponding band projectors and is the Fermi-Dirac distribution. We have checked that the off-diagonal part of the inverse of the vertex function () is negligible because of frustration in the momentum space. We denote the diagonal part as . is actually connected to the static spin-triplet Cooper pair susceptibility of the non-interacting system:

(19) |

The critical temperature is determined by the condition . When the superconductivity pairing is stable and the static spin-triplet Cooper pair response indicates the profile of the triplet Cooper pairs’ condensate as plotted in Fig. 3 in the main text.

### a.3 Coherence length

The FFLO superconductivity coherence length takes the form . The calculation of coherence length suggests that the Cooper pair pairing is short ranged around quarter-filling. In Fig. 7, we plot versus (a) and versus (b). This suggests that around quarter-filling and the electron pairing may involves longer range pairing other than the nearest-neighbor pairing plotted in Fig. 2 in the main text.

### a.4 Bloch theorem analysis for FFLO ground state wave function

Here, we analyze the momentum sectors of the low lying states when the system in Eq. 1 is doped with a single Cooper pair. The primitive wave vectors in the direct space are denoted respectively as (i=1, 2) with , . We numerically diagonalized the doped system and plot the energy levels in the different momentum sectors. On the discretized system of plaquettes on torus, we will have thereafter momentum sectors in the basis of and momentum sectors in the basis of , which leads to a total of sectors. To represent the 2D numerical spectra in 1D, we plot the energy levels as a function of in which is the number of plaquettes in the x direction. We can apply the Bloch theorem to analyze the footprints of the FFLO superconductivity. If we denote the three degenerate ground states with one Cooper pair as , and for the spin-triplet x, y and z, we have:

(20) |

We have: in which is the momentum corresponding to the momentum sector of the ground state, we can therefore identify the three ground state Ansätze , and : In the system with plaquettes, and need to be even numbers so that the FFLO superconductivity is not frustrated. The three momentum sectors for the ground state wave function will be in , and . For a system of odd number of Cooper pairs, the total momentum equals the FFLO Cooper pair momenta are the same as the scenario with only one Cooper pair since and for a system of even number of Cooper pairs, the total momentum is zero . This different behavior for ground state momentum sector between odd and even number of Cooper pairs is shown in Figs. 6 in the main text, which distinguishes the FFLO superconductivity from other spatially modulated orders like spin or charge density wave.

### a.5 Stability of the FFLO phase when

We can also compute the pairing order parameter by numerical computation of the expectation value of different order parameters in the system of plaquettes on torus as indicated in Fig. 2:

(21) |

in which the subscripts N and N+2 are the numbers of electrons for the wave functions. is the triplet pairing operator given in Eq. 3 and . Using this procedure, we have calculated the superconductor pairing order parameter on different links for the plaquettes on torus. The ground state for the half-filling system () is unique while the ground state for the two electron doped system is three times degenerate. Using Eq. 21, we obtain three set of order parameters and for the numerical leading order (other order parameters are at least ten times smaller), we have respectively:

(22) |

in which the numbering of the sites is represented in Fig. 2. It is worth noting that the spin-triplet pairing operators are antisymmetric: . The expectation values of the superconductor pairing order parameter are shown in Fig. 8, in which we parametrize as The numerical results confirm the emergence of triplet superconductivity with phase modulation for the pairing order parameter in the direct space when . We have checked that the charge or spin density order is negligible.

We have also studied the inverse of the Cooper pair vertex function for the spin-triplet pairing () in the limit . The hopping term in Eq. 1 opens a gap for the band structure and the linear dispersion relation becomes quadratic, which makes the susceptibility finite in the limit of zero temperature and the susceptibility becomes lower when one turns on gradually the hopping term as shown in Fig. 9. The FFLO phase appears as long as the energy related to the critical temperature is bigger than the gap of the free electron system around quarter-filling opened by the term i.e. . When are large enough, the spin-singlet state will become gradually energetically favorable, which triggers the entanglement of the four Cooper pairs expressed in Eq. 3 with a term proportional to in the Landau expansion.

### References

- P.W. Anderson, Science 235, 4793 (1987).
- F.C. Zhang, T.M. Rice, Phys. Rev. B 37, 3759 (1988).
- M. Sigrist and K. Ueda, Rev. Mod. Phys. 63, 239, (1991).
- P.W. Anderson, P.A. Lee, M. Randeria, T.M. Rice, N. Trivedi, and F.C. Zhang, J. Phys. Condens. Matter 16, R755 (2004).
- K. Le Hur and T. M. Rice, Annals of Physics 324 1452 (2009).
- C. N. Yang, Phys. Rev. Lett 63, 2144 (1989).
- P. Fulde and R. A. Richard, Phys. Rev. 135: A550 (1964).
- A. I. Larkin, Y. N. Ovchinnikov, Zh. Eksp. Teor. Fiz. 47: 1136 (1964).
- A. I. Larkin, Y. N. Ovchinnikov, Sov. Phys. JETP 20: 762 (1965).
- A. Bianchi, R. Movshovich, C. Capan, A. Lacerda, P. G. Pagliuso, and J. L. Sarrao, Phys. Rev. Lett. 91, 187004 (2003).
- Yean-an Liao, et al. Nature 467, 567-569 (2010).
- Martin W. Zwierlein, André Schirotzek, Christian H. Schunck and Wolfgang Ketterle, Science 311 (5760), 492-496 (2006).
- C. Mora and R. Combescot, Phys. Rev. B 71, 214504 (2005).
- F. Chevy, C. Mora, Reports on Progress in Physics 73, 11 (2010).
- Z. Cai, Y. Wang, and C. Wu, Phys. Rev. A 83, 063621 (2001).
- T.K. Koponen, T. Paananen, J.-P. Martikainen and P. Torma, Phys. Rev. Lett. 99, 120403 (2007).
- I. Vasić, A. Petrescu, K. Le Hur, and W. Hofstetter, Phys. Rev. B 91, 094502 (2015).
- A. M. Berridge, A. G. Green, S. A. Grigera, and B. D. Simons, Phys. Rev. Lett. 102, 136404 (2009).
- W. Witczak-Krempa and Y. B. Kim, Phys. Rev. B 85, 045124 (2012).
- A. Shitade, H. Katsura, J. Hunes, X.-L. Qi, S.-C. Zhang and N. Nagaosa, Phys. Rev. Lett. 102 256403 (2009).
- J.G. Rau, E. K. Lee, H.-Y. Kee, Arxiv: 1507.06323.
- D. A. Pesin and L. Balents, Nature Phys. 6, 376 (2010).
- A. Kitaev, Annals of Physics 321, 2-111 (2006).
- G.B. Halász, J.T. Chalker and R. Moessner, Phys. Rev. B 90, 035145 (2014).
- J. Chaloupka, G. Jackeli and G. Khaliullin, Phys. Rev. Lett. 105, 027204 (2010).
- G.Jackeli and G.Khaliullin, Phys. Rev. Lett. 102, 017205 (2009).
- Y. Singh and P. Gegenwart, Phys. Rev. B 82, 064412 (2010).
- Y. Singh, S. Manni, J. Reuther, T. Berlijn, R. Thomale, W.Ku, S. Trebst, and P. Gegenwart, Phys. Rev. Lett. 108, 127203 (2012)
- J. Reuther, R. Thomale, S. Trebst, Phys. Rev. B 84, 100406 (R) (2011).
- S.-H. Chun et al., Nature Physics, accepted (2015), arXiv:1504.03618.
- T. M. Rice, K. Ueda, Phys. Rev. Lett 55, 995 (1985).
- T. M. Rice, K. Ueda, Phys. Rev. Lett 55, 2093 (1985).
- B. Edegger, V. N. Muthukumar and C. Gros, Advances in Physics, Vol. 56, No. 6, November-December 2007, 927-1033
- Y. Z. You, I. Kimchi, A. Vishwanath, Phys. Rev. B 86, 085145 (2012).
- T. Hyart, A.R. Wright, G. Khaliullin, B. Rosenow, Phys. Rev. B. 85, 140510(R) (2012).
- G. Cao, T. F. Qi, L. Li, J. Terzic, V.S. Cao, S. J. Yuan, M. Tovar, G. Murthy, and R. K. Kaul, Phys. Rev. B 88, 220414(R) (2013).
- S. R. Hassan, S. K. Goyal, R. Shankar, D. Sénéchal, Phys. Rev. B 88, 045301 (2013).
- L.-M. Duan, E. Demler and M. D. Lukin, Phys. Rev. Lett. 91, 090402 (2003).
- R. Nandkishore, L. Levitov, A. Chubukov, Nature 8, 158-163 (2012).
- A. M. Black-Schaffer, C. Honerkamp, J. Phys.: Condens. Matter 26, 423201 (2014).
- A. J. Leggett, Rev. Mod. Phys. 47, 331 (1975).
- T. Zibold, V. Corre, C. Frapolli, J. Dalibard and F. Gerbier, Arxiv: 1506.06176.
- V. Corre, T. Zibold, C. Frapolli, L. Shao, J. Dalibard and F. Gerbier, EuroPhyiscs Letters 110, 26001 (2015).
- A.Altland, B.Simons, Condensed Matter Field Theory, Cambridge University Press (2006).
- The superconducting instability is most significant around the symmetry centers of the Fermi surface with momentum since, for any , electron pairs with respectively momentum and are both located on the Fermi surface therefore the population of electrons forming Cooper pairs is the largest.
- See Supplemental Material