# Mott transition and magnetism of the triangular-lattice Hubbard model

with next-nearest-neighbor hopping

###### Abstract

The variational cluster approximation is used to study the isotropic triangular-lattice Hubbard model at half filling, taking into account the nearest-neighbor () and next-nearest-neighbor () hopping parameters for magnetic frustrations. We determine the ground-state phase diagram of the model. In the strong correlation regime, the 120 Néel and stripe ordered phases appear, and a nonmagnetic insulating phase emerges in between. In the intermediate correlation regime, the nonmagnetic insulating phase expands to a wider parameter region, which goes into a paramagnetic metallic phase in the weak correlation regime. The critical phase boundary of the Mott metal-insulator transition is discussed in terms of the van Hove singularity evident in the calculated density of states and single-particle spectral function.

###### pacs:

71.10.Fd 75.10.Jm 71.30.+h 75.10.-b## I Introduction

The physics of geometrical frustration in strongly correlated electron systems has long attracted much attention anderson (); lee (); balents (). In particular, possible absence of magnetic long-range orders at zero temperature in the Heisenberg and Hubbard models defined on frustrated lattices, or the realization of a spin liquid phase as an exotic state of matter, has been one of the major issues in this field. The Mott metal-insulator transition is also a fundamental phenomenon in the field of strongly correlated electron systems mott (); imada (), which has attracted much experimental and theoretical interest as well. As one of the simplest models with geometrical frustration and Mott transition, we therefore study the Hubbard model at half filling defined on the triangular lattice in this paper, where not only the nearest-neighbor hopping parameters but also the next-nearest-neighbor ones are included.

Much effort has so far been devoted in the study of the triangular-lattice Hubbard model with anisotropic nearest-neighbor hopping parameters morita (); sahebsara (); watanabe (); ohashi (); tocchio1 (); tocchio2 (); yamada1 (); laubach (); misumi (), which was motivated by experimental findings of possible spin liquid states in some organic Mott insulators such as -(ET)Cu(CN) komatsu (); shimizu (); kurosaki (); manna () and EtMeSb[Pd(dmit)] itou (); yamashita (). The triangular-lattice Heisenberg model with the anisotropic exchange interactions has also been studied to find a variety of ordered phases such as Néel and spiral orders, as well as the quantum disordered (or spin liquid) phases in-between weihong (); hauke1 (); hauke2 ().

However, to the best of our knowledge, the isotropic triangular-lattice Hubbard model with both the nearest-neighbor () and next-nearest-neighbor () hopping parameters has not yet been addressed, the study of which will therefore provide useful information on the physics of magnetic frustrations and Mott metal-insulator transition in strongly correlated electron systems.

The isotropic Heisenberg model with the nearest-neighbor () and next-nearest-neighbor () exchange interactions, which may be derived by the second-order perturbation of the above-mentioned Hubbard model in the strong correlation limit, has on the other hand been studied much in detail, mostly from the theoretical point of view ma (). In the classical Heisenberg model where the spins are treated as classical vectors, it is known that a single phase transition occurs at between the three-sublattice 120 Néel ordered state and an infinitely degenerate four-sublattice magnetically ordered states jolicoeur (). This degeneracy is lifted by quantum fluctuations, thereby selecting a two-sublattice stripe ordered state through the so-called “order-by-disorder” mechanism jolicoeur (); chubukov (); deutscher (); lecheminant (). One may then expect in the corresponding quantum Heisenberg model that an intermediate phase can appear near the classical critical point at , for which many studies have been carried out to predict that the nonmagnetic disordered phase bordered by the 120 Néel ordered phase at and the stripe ordered phase at actually emerges. In particular, recent studies actually predict the emergence of either a gapless or gapped spin liquid phase in this intermediate region manuel (); mishmash (); kaneko (); li2 (); zhu (); hu2 (); iqbal (). These results of the Heisenberg model may be compared with those of our Hubbard model in the strong correlation limit (as we will see below).

In this paper, motivated by the above developments in the field, we will study the triangular-lattice Hubbard model at half filling with the isotropic nearest-neighbor and next-nearest-neighbor hopping parameters in its entire interaction strength. We use the variational cluster approximation (VCA), one of the quantum cluster methods based on the self-energy functional theory (SFT) potthoff1 (); potthoff2 (); dahnken (); potthoff3 (); senechal (), which enables us to take into account the quantum fluctuations of the model with geometrically frustrated spin degrees of freedom. We thereby calculate the grand potential of the system as a function of the Weiss fields for spontaneous symmetry breakings; here, we take the 120 Néel and stripe magnetic orders and evaluate the order parameters and critical interaction strengths. We also calculate the charge gap as well as the density of states (DOS) and single-particle spectral function using the cluster perturbation theory (CPT) senechal () and determine the ground-state phase diagram of the model in its entire parameter region.

We will thereby show that in the strong correlation regime the 120 Néel and stripe ordered phases appear, and in-between, the nonmagnetic insulating phase caused by the quantum fluctuations in the frustrated spin degrees of freedom emerges, in agreement with the Heisenberg model studies. We will also show that in the intermediate correlation regime the nonmagnetic insulating phase expands to wider parameter regions located around and , which go into a paramagnetic metallic phase in the weak correlation regime via the second-order Mott transition. The characteristic behavior of the critical phase boundary of the Mott transition is discussed in terms of the van Hove singularity appearing in the calculated DOS and single-particle spectral function.

The rest of the paper is organized as follows. In Sec. II, we introduce the model and discuss the method of calculation briefly. In Sec. III A, we present our results obtained in the strong correlation regime and compare them with those of the Heisenberg model. In Sec. III B, we present our results obtained in the intermediate to weak correlation regime and discuss the phase diagram of our model. The critical phase boundary of the Mott transition is also discussed. A summary of the paper is given in Sec. IV.

## Ii Model and Method

We consider the triangular-lattice Hubbard model [see Fig. 1(a)] defined by the Hamiltonian

(1) |

where () creates (annihilates) an electron with spin at site , and . indicates the nearest-neighbor bonds with the hopping parameter and indicates the next-nearest-neighbor bonds with the hopping parameter . We consider the parameter region , including two limiting cases, (isotropic triangular lattice) and . is the on-site Coulomb repulsion between two electrons and is the chemical potential maintaining the system at half filling.

In the large- limit, this model may be mapped onto the triangular-lattice Heisenberg model of spin-1/2 defined by the Hamiltonian

(2) |

with the exchange coupling constants of and for the nearest-neighbor and next-nearest-neighbor bonds, respectively. The spin operator is given by /2 with the vector of Pauli matrices . The results obtained for the Hubbard model [Eq. (1)] in the strong correlation regime are compared with those of the Heisenberg model [Eq. (2)].

Let us describe the VCA briefly, which is a many-body variational method based on the SFT, where the grand potential of the system is formulated as a functional of the self-energy potthoff1 (); potthoff2 (); dahnken (). The ground state of the original system in the thermodynamic limit can thus be obtained via the calculation of the grand potential of the system with the exact self-energy. Then, in the VCA, restricting the trial self-energy to the self-energy of the reference system , we obtain the approximate grand potential as

(3) |

where is the grand potential of the reference system, and and are the noninteracting Green’s functions of the original and reference systems, respectively. The Hamiltonian of the reference system is defined below. Note that the short-range correlations within the clusters of the reference system are taken into account exactly. See Refs. potthoff3, ; senechal, for recent reviews of the method.

The advantage of the VCA is that the spontaneous symmetry breaking can be treated within the framework of the theory, where we introduce the Weiss fields as variational parameters. In the present case, the Hamiltonian of the reference system is taken as with the Weiss fields

(4) | |||

(5) | |||

(6) |

where and are the strengths of the Weiss fields for the 120 Néel and stripe ordered states, respectively. For the Néel order, the unit vectors are rotated by 120 to each other, where () is the sublattice index of site . For the stripe order, the wave vectors can be taken equivalently as either , , or . The variational parameters are optimized on the basis of the variational principle, i.e., , for each magnetic order, where the solution with corresponds to the ordered state.

In our VCA calculations, we use the 12-site cluster shown in Fig. 2 as the reference system. This is the best appropriate and feasible choice of the reference cluster because we can treat the two-sublattice order (stripe order) with an equal number of up- and down-spin electrons and the three-sublattice order (120 Néel order) with an equal number of the three sublattice sites , 2 and 3. The cluster-size and cluster-shape dependences of our results are discussed in Appendix. Note that longer period phases such as the spiral phase mentioned in a different system tocchio2 () cannot be treated in the present approach; in our analysis, we fix the pitch angle of the spiral order to be 120 (or the three-sublattice of ) even for . The charge orderings discussed in the extended Hubbard model with intersite Coulomb repulsions tocchio3 () are also neglected. To our knowledge, no other orders have been predicted in the present triangular-lattice Hubbard and Heisenberg models.

## Iii Results of calculation

### iii.1 Strong correlation regime

First, let us discuss the strong correlation regime, . We calculate the ground-state energies (per site) and magnetic order parameters defined as for the 120 Néel order and for the stripe order, where stands for the ground-state expectation value. The results are shown in Fig. 3, where we find three phases: the 120 Néel ordered phase around , the stripe ordered phase around , and the nonmagnetic disordered phase in-between.

At , the 120 Néel ordered state has the lowest energy and with increasing it approaches the energy of the nonmagnetic disordered state gradually. Then, at , the 120 Néel ordered state disappears continuously. The calculated order parameter also indicates the continuous (or second-order) phase transition. On the other hand, at , the stripe ordered state has the lowest energy and, with decreasing , the energy of stripe order crosses to that of the nonmagnetic state at , indicating the discontinuous (or first-order) transition between the stripe and disordered phases. The calculated order parameter also disappears discontinuously at .

These results may be compared with the previous studies on the - triangular-lattice Heisenberg model manuel (); mishmash (); kaneko (); li2 (); zhu (); hu2 (). The transition point between the 120 Néel and nonmagnetic phases has been estimated to be , which corresponds to of our Hubbard model parameters. A reasonable agreement is thus obtained. The transition point between the stripe and nonmagnetic phases has also been estimated to be , which corresponds to of our Hubbard model parameters. We again find a reasonable agreement with our estimation. The orders of the phase transitions, i.e., the second-order for the 120 Néel phase and the first-order for the stripe phase, are also in agreement with the previous study of the Heisenberg model kaneko (). We may point out that the strong quantum fluctuations in the frustrated spin degrees of freedom causes this nonmagnetic phase because the classical spin model predicts either the 120 Néel or four-sublattice ordered phase without any intermediate nonmagnetic phases jolicoeur (); chubukov (); deutscher (); lecheminant ().

### iii.2 Intermediate to weak correlation regime

Next, let us discuss the intermediate to weak correlation regime . We here calculate the total energies, order parameters, and charge gaps of the model, as well as the grand potential as a function of the Weiss fields, and summarize them as the ground-state phase diagram in the parameter space , as shown in Fig. 4. We find four phases: the 120 Néel and stripe ordered phases at large , which are continuous to the phases at discussed above, and nonmagnetic insulating phase in-between, as well as the paramagnetic metallic phase in the weak correlation regime. In the intermediate correlation regime, the nonmagnetic insulating phase expands to wider parameter regions, which are around and around . We note that the presence of the nonmagnetic insulating phase around is in agreement with previous studies of the triangular-lattice Hubbard model at sahebsara2 (); yoshioka1 (); yoshioka2 (); yamada1 (); laubach ().

The calculated order parameters of the 120 Néel and stripe phases are shown in Fig. 5 as a function of for several values of . We find that the transition to the stripe ordered phase is continuous, irrespective of , up to a large value of , but it changes to the discontinuous transition as seen in Fig. 3 at . We also find that the transition to the 120 Néel ordered phase is discontinuous at for but it is continuous for larger values of . The transition at is also continuous (see Fig. 3). These behaviors are observed also in the calculated Weiss-field dependence of the grand potentials of our model.

The charge gap is evaluated from the total number of electrons as a function of (see Fig. 5) to examine the Mott metal-insulator transition of the system. We find that the transition is continuous (or second-order) and the phase boundary is located around , as shown in Fig. 4. We note that the phase boundary decreases (shifts to a lower side) with increasing up to , but it increases for larger values of . This behavior is in contrast to that of the square-lattice Hubbard model with the next-nearest-neighbor hopping parameters, where a monotonous increase in the critical interaction strength is observed mizusaki (); nevidomskyy (); yamada2 (), which is due to the monotonous increase in the band width of the model.

To find out the origin of this behavior, we calculate the DOS and single-particle spectral function in the paramagnetic state of the system using the CPT, which are defined as

(7) | |||

(8) |

with the CPT Green’s function senechal ()

(9) |

where we define the matrices for the cluster of size as with . The exact Green’s function of the reference system is given by

(10) |

where and are the ground state and ground state energy of .

The calculated results for the DOS and single-particle spectral function of our model are shown in Figs. 6 and 7, respectively. We find that the sharp peak appeared above the Fermi level at , which is caused by the van Hove singularity in the triangular lattice, shifts to the lower energy side with increasing , and at , the peak position coincides with the Fermi level (see Fig. 6). This situation of the high DOS at the Fermi level is energetically unstable fazekas (), so that the band gap opens to gain in the band energy in the presence of the Hubbard interaction . With further increasing , the peak shifts to the higher energy side again. The Hubbard band gap is then the largest at as seen in Figs. 5 and 6. This singularity is also seen in the single-particle spectral function as the presence of the flat-band region around the K point of the Brillouin zone (see Fig. 7). This behavior thus explains why the critical interaction strength becomes small at around .

To confirm the absence of any magnetic instability in our model in the weak correlation regime, we here calculate the generalized susceptibility (or Lindhard function) in the noninteracting limit, which is defined as

(11) |

where is the corresponding noninteracting band dispersion and is the Fermi function. The calculated results at temperature are shown in Fig. 8, where we find that, in accordance with the absence of significant Fermi-surface nesting features, no singular behaviors actually appear in , indicating the absence of magnetic long-range orders in the weak correlation limit. This result supports the validity of our phase diagram shown in Fig. 4 in the weak correlation regime.

## Iv Summary

We have studied the Mott metal-insulator transition and magnetism of the triangular-lattice Hubbard model at half filling in the entire region of the interaction strength, taking into account the next-nearest-neighbor hopping parameters for the effects of magnetic frustrations. We have employed the method of VCA based on the SFT, which has not been used for the present purposes. We have thereby calculated the grand potential of the system as a function of the Weiss fields for the 120 Néel and stripe magnetic orders, and have determined the order parameters. We have also calculated the DOS and single-particle spectral function as well as the charge gap of the system. These results have been summarized as the ground-state phase diagram of the system.

We have found four phases: In the strong correlation regime, there appear (i) the 120 Néel ordered phase in a wide parameter region around and (ii) the stripe ordered phase in a wide parameter region around , and in-between, (iii) the nonmagnetic insulating phase caused by the quantum fluctuations in the geometrically frustrated spin degrees of freedom emerges. The obtained phase boundaries in the strong correlation limit have been compared with those of the corresponding Heisenberg model to find a reasonable agreement. The orders of the phase transitions of the two magnetically ordered phases have also been determined. In the intermediate correlation regime, the nonmagnetic insulating phase expands to a wider parameter region of . Then, decreasing the interaction strength further, the system turns into (iv) the paramagnetic metallic phase in the weak correlation regime via the second-order Mott metal-insulator transition. The characteristic behavior of the critical phase boundary of the Mott transition has also been discussed in terms of the shift in the van Hove singularity due to the presence of , as seen in the calculated DOS and single-particle spectral function.

We suggest that the phase diagram obtained here may contain different types of nonmagnetic insulator (or spin liquid) states depending on the region in the parameter space. The characterization of the states is however beyond the scope of the VCA approach based on the self-energy (or single-particle Green’s function), for which we hope that our results will encourage future studies.

###### Acknowledgements.

We thank S. Miyakoshi for enlightening discussions and K. Seki for careful reading of our manuscript. This work was supported in part by a Grant-in-Aid for Scientific Research (No. 26400349) from JSPS of Japan. T. K. acknowledges support from the JSPS Research Fellowship for Young Scientists.*

## Appendix A Cluster dependence of the phase boundaries

In the VCA, or in any other quantum cluster methods, we can in principle calculate the
physical quantities in the thermodynamic limit, but the calculated results necessarily
depend on the size and shape of the solver cluster. Thus, the choice of the solver
cluster is important in the present approach.
In the main text, we have chosen the 12-site cluster shown in Fig. 2, which is the
best appropriate one because it is first of all computationally feasible and also because it fits
with both the three-sublattice 120 order and two-sublattice stripe order without
introducing unnecessary frustrations.
However, it seems instructive to check the cluster-size and cluster-shape dependences
of our results presented in the main text.

Here, we choose several clusters [see Fig. 9(b)] that fit either with the 120
order or with the stripe order, and we calculate the phase boundaries to check the solver
cluster dependence of the ground-state phase diagram shown in Fig. 4.
The calculated results for the phase boundaries are shown in Fig. 9(a).
We thus find that the phase boundary between the 120 Néel ordered and
nonmagnetic insulating phases is located around in an intermediate
to large region and that the phase boundary between the stripe ordered
and nonmagnetic insulating phases is located around in an intermediate
to large region, irrespective of the appropriate choices of the solver cluster.
We also find that the phase boundary of the Mott metal-insulator transition is located
around with a minimum at , irrespective of the
choices of the solver cluster.

## References

- (1) P. Anderson, Mater. Res. Bull. 8, 153 (1973).
- (2) L. Balents, Nature (London) 464, 199 (2010).
- (3) P. A. Lee, N. Nagaosa, and X. -G. Wen, Rev. Mod. Phys. 78, 17 (2006).
- (4) N. Mott, Metal-Insulator Transitions (Taylor and Francis, London, UK, 1990).
- (5) M. Imada, A. Fujimori, and Y. Tokura, Rev. Mod. Phys. 70, 1039 (1998).
- (6) H. Morita, S. Watanabe, and M. Imada, J. Phys. Soc. Jpn. 71, 2109 (2002).
- (7) P. Sahebsara and D. Sénéchal, Phys. Rev. Lett. 97, 257004 (2006).
- (8) T. Watanabe, H. Yokoyama, Y. Tanaka, and J. Inoue, Phys. Rev. B 77, 214505 (2008).
- (9) T. Ohashi, T. Momoi, H. Tsunetsugu, and N. Kawakami, Phys. Rev. Lett. 100, 076402 (2008).
- (10) L. F. Tocchio, A. Parola, C. Gros, and F. Becca, Phys. Rev. B 80, 064419 (2009).
- (11) L. F. Tocchio, H. Feldner, F. Becca, R. Valenti, and C. Gros, Phys. Rev. B 87, 035143 (2013).
- (12) A. Yamada, Phys. Rev. B 89, 195108 (2014).
- (13) M. Laubach, R. Thomale, C. Platt, W. Hanke, and G. Li, Phys. Rev. B 91, 245125 (2015).
- (14) K. Misumi, T. Kaneko, and Y. Ohta, J. Phys. Soc. Jpn., in press (2016).
- (15) T. Komatsu, N. Matsukawa, T. Inoue, and G. Saito, J. Phys. Soc. Jpn. 65, 1340 (1996).
- (16) Y. Shimizu, K. Miyagawa, K. Kanoda, M. Maesato, and G. Saito, Phys. Rev. Lett. 91, 107001 (2003).
- (17) Y. Kurosaki, Y. Shimizu, K. Miyagawa, K. Kanoda, and G. Saito, Phys. Rev. Lett. 95, 177001 (2005).
- (18) R. S. Manna, M. de Souza, A. Brühl, J. A. Schlueter, and M. Lang, Phys. Rev. Lett. 104, 016403 (2010).
- (19) T. Itou, A. Oyamada, S. Maegawa, M. Tamura, and R. Kato, Phys. Rev. B 77, 104413 (2008).
- (20) M. Yamashita, N. Nakata, Y. Senshu, M. Nagata, H. M. Yamamoto, R. Kato, T. Shibauchi, and Y. Matsuda, Science 328, 1246 (2010).
- (21) Z. Weihong, R. H. McKenzie, and R. R. P. Singh, Phys. Rev. B 59, 14367 (1999).
- (22) P. Hauke, T. Roscilde, V. Murg, J. Cirac, and R. Schmied, New J. Phys. 13, 075017 (2011).
- (23) P. Hauke, Phys. Rev. B 87, 014415 (2013).
- (24) A rare exception is BaCoSbO, which was reported to be an almost perfect realization of a spin-1/2 equilateral triangular-lattice antiferromagnet. See, e.g., J. Ma, Y. Kamiya, T. Hong, H. B. Cao, G. Ehlers, W. Tian, C. D. Batista, Z. L. Dun, H. D. Zhou, and M. Matsuda, Phys. Rev. Lett. 116, 087201 (2016).
- (25) T. Jolicoeur, E. Dagotto, E. Gagliano, and S. Bacci, Phys. Rev. B 42, 4800 (1990).
- (26) A. V. Chubukov and T. Jolicoeur, Phys. Rev. B 46, 11137 (1992).
- (27) R. Deutscher and H. U. Everts, Z. Phys. B 93, 77 (1993).
- (28) P. Lecheminant, B. Bernu, C. Lhuillier, and L. Pierre, Phys. Rev. B 52, 6647 (1995).
- (29) L. O. Manuel and H. A. Ceccatto, Phys. Rev. B 60, 9489 (1999).
- (30) R. V. Mishmash, J. R. Garrison, S. Bieri, and C. Xu, Phys. Rev. Lett. 111, 157203 (2013).
- (31) R. Kaneko, S. Morita, and M. Imada, J. Phys. Soc. Jpn. 83, 093707 (2014).
- (32) P. H. Y. Li, R. F. Bishop, and C. E. Campbell, Phys. Rev. B 91, 014426 (2015).
- (33) Z. Zhu and S. R. White, Phys. Rev. B 92, 041105 (2015).
- (34) W.-J. Hu, S.-S. Gong, W. Zhu, and D. N. Sheng, Phys. Rev. B 92, 140403 (2015).
- (35) Y. Iqbal, W.-J. Hu, R. Thomale, D. Poilblanc, and F. Becca, Phys. Rev. B 93, 144411 (2016).
- (36) M. Potthoff, M. Aichhorn, and C. Dahnken, Phys. Rev. Lett. 91, 206402 (2003).
- (37) M. Potthoff, Eur. Phys. J. B 32, 429 and 36 335 (2003).
- (38) C. Dahnken, M. Aichhorn, W. Hanke, E. Arrigoni, and M. Potthoff, Phys. Rev. B 70, 245110 (2004).
- (39) M. Potthoff, in Strongly Correlated Systems–Theoretical Methods, Vol. 171 of Springer Series in Solid-State Sciences (Springer-Verlag, Berlin Heidelberg, 2012), Chap. 10, pp. 303–339.
- (40) D. Sénéchal, in Strongly Correlated Systems–Theoretical Methods, Vol. 171 of Springer Series in Solid-State Sciences (Springer-Verlag, Berlin Heidelberg, 2012), Chap. 8, pp. 237–270.
- (41) L. F. Tocchio, C. Gros, X.-F. Zhang, and S. Eggert, Phys. Rev. Lett. 113, 246405 (2014).
- (42) P. Sahebsara and D. Sénéchal, Phys. Rev. Lett. 100, 136402 (2008).
- (43) T. Yoshioka, A. Koga, and N. Kawakami, Phys. Rev. Lett. 103, 036401 (2009).
- (44) T. Yoshioka, A. Koga, and N. Kawakami, Phys. Stat. Sol. (b) 247, 635 (2010).
- (45) T. Mizusaki and M. Imada, Phys. Rev. B 74, 014421 (2006).
- (46) A. H. Nevidomskyy, C. Scheiber, D. Sénéchal, and A. -M. S. Tremblay, Phys. Rev. B 77, 064427 (2008).
- (47) A. Yamada, K. Seki, R. Eder, and Y. Ohta, Phys. Rev. B 88, 075114 (2013).
- (48) P. Fazekas, Lecture Notes on Electron Correlation and Magnetism, Vol. 5 of Series in Modern Condensed Matter Physics (World Scientific, Singapore, 1999).