A phase diagram for a topological Kondo insulating system

# A phase diagram for a topological Kondo insulating system

Minh-Tien Tran, Tetsuya Takimoto, and Ki-Seok Kim Asia Pacific Center for Theoretical Physics, POSTECH, Pohang, Gyeongbuk 790-784, Republic of Korea
Institute of Physics, Vietnamese Academy of Science and Technology, P.O.Box 429, 10000 Hanoi, Vietnam
Department of Physics, POSTECH, Pohang, Gyeongbuk 790-784, Republic of Korea
July 7, 2019
###### Abstract

The discovery of topological insulators in non-interacting electron systems has motivated the community to search such topological states of matter in correlated electrons both theoretically and experimentally. In this paper we investigate a phase diagram for a topological Kondo insulating system, where an emergent “spin”-dependent Kondo effect gives rise to an inversion for heavy-fermion bands, responsible for a topological Kondo insulator. Resorting to the U(1) slave-boson mean-field analysis, we uncover an additional phase transition inside the Kondo insulating state in two dimensions, which results from the appearance of the topological Kondo insulator. On the other hand, we observe that the Kondo insulating state is distinguished into three insulating phases in three dimensions, identified with the weak topological Kondo insulator, the strong topological Kondo insulator, and the normal Kondo insulator, respectively, and classified by Z topological indices. We discuss the possibility of novel quantum criticality between the fractionalized Fermi liquid and the topological Kondo insulator, where the band inversion occurs with the formation of the heavy-fermion band at the same time.

## I Introduction

An effective field theory approach has been playing an important role in predicting novel quantum states of matter. They are well developed chiral anomaly for quantum number fractionalization, Chiral_Anomaly () parity anomaly for quantum Hall effect, Parity_Anomaly () and an SU(2) global anomaly Witten_Anomaly () for topological insulators in three dimensions Ryu_TI (). However, the field theory approach is not enough to search such quantum matter in real materials. It is necessary to construct or find the corresponding lattice model, which can result in our wishful effective field theory at low energies. In particular, the lattice model can be solved almost exactly based on numerical simulations, helping us understand the connection between the lattice model and effective field theory and verifying the possibility of novel quantum phenomena.

The Su-Schrieffer-Heeger model, which describes one dimensional electrons coupled with lattices, confirms the existence of fractional electric charge, carried by domain wall solitons. Su_Schrieffer_Heeger_Model () The Haldane model, which describes two dimensional electrons with a next nearest neighbor complex hopping parameter on the honeycomb lattice, explains the integer quantum Hall effect without Landau levels formed from external magnetic field, Haldane_Model () where each Dirac band carries a nontrivial topological quantum number (Chern number), identified with the quantized Hall conductance. TKNN () Recently, the Haldane model has been generalized into the case with time reversal invariance, where the spin-orbit coupling serves an effective magnetic flux, oppositely assigned to spin up and down electrons, thus regarded as two copies of Haldane models. The Kane-Mele model has proposed an interesting insulating state, called a topological insulator, where the quantum spin Hall effect appears when the -component of spin quantum number is preserved, but generically classified by the Z topological quantum number with spin non-conserving terms. Kane_Mele_Model () The concept of the topological insulator was extended into the three dimensional case, 3D_TI_Z2index_FuKane (); 3D_TI_Z2index_Balents (); 3D_TI_Z2index_Roy () characterized by the Z topological index which counts the number of band inversions with modular two. Odd number of band inversions cause odd number of Dirac bands, identified with normalizable fermion zero modes localized at each two dimensional surface, where one of them at least is protected against time reversal invariant perturbations due to a topological origin.

Considering that these models describe non-interacting electrons basically, an immediate and important question is on the role of electron correlations in such topological states of matter. Balents_MIT_SO () Actually, this direction of research has been performed intensively, in particular, focusing on the emergence of gapped spin liquids, Balents_Review () where spinons with fractional spin quantum number appear as elementary excitations. The quantum dimer model on the triangular lattice has confirmed the existence of a short-ranged resonating valence bond state, identified with Z spin liquid. QDM_Z2SL () This gapped spin liquid state has been claimed to appear in strongly correlated electrons on the honeycomb lattice. Dirac_Z2SL () Such a spirit has been also realized in strongly correlated spinless fermions, Fulde_Fractional_Charge () where charge fractionalization occurs in geometrically frustrated lattices. Artificial spin models but exactly solvable, represented as the Kitaev model, Kitaev_Model () have been investigated, proving the existence of topologically non-trivial quantum states of matter, for example, a non-abelian fractional quantum Hall state, which allows Majorana fermions. Moore_Green () Such exotica has been also pursued near quantum criticality, referred as deconfined quantum criticality, where an effective field theory approach had suggested the emergence of spin quantum number fractionalization at an antiferromagnet to valence bond solid quantum critical point, DQCP () and numerical simulations for an extended Heisenberg model on the square lattice confirmed this scenario with some modifications. Sandvick (); Balents_etal ()

Recently, the fractional quantum Hall effect has been argued to appear in a certain type of lattice models, whose characteristic feature is the existence of an almost flat band with a non-trivial Chern number. FQHE_Lattice_Model () Electron correlations for this partially filled flat band with the Chern number were demonstrated to result in the fractional quantum Hall phase. An immediate and interesting question is how to construct the lattice model, which shows the so called fractional topological insulator. An effective field theory approach argued the possibility of the fractional topological insulator, FTI_EFT_Theta1 (); FTI_EFT_Theta2 () characterized by the fact that the surface Dirac fermion carries a fractional electric charge, which should be distinguished from the fractionalized “normal” insulator, not allowing gapless surface modes. A hard-core boson model was proposed on the diamond lattice, where the boson is assumed to fractionalize into two fermions with fractional charge, forming a topological band insulator for such fractionalized fermions and displaying the fractional magnetoelectric effect. FTI_Lattice_Model () Introducing electron correlations into the Kane-Mele model, Kane_Mele_Hubbard_Model1 (); Kane_Mele_Hubbard_Model2 () an interesting spin liquid state was proposed to exhibit an integer quantum spin Hall effect in the case of conservation.

In this paper we investigate an effective Anderson lattice model for the so called topological Kondo insulator, regarded as one of the most studied models, particulary, for non-Fermi liquid physics near heavy fermion quantum criticality. HFQCP_Review () The existence of the topological Kondo insulator was pointed out recently, where an emergent “spin”-dependent Kondo effect gives rise to an inversion for heavy-fermion bands, which turns out to be responsible for the topological Kondo insulator. TKI_Coleman () Resorting to the U(1) slave-boson mean-field analysis, we uncover an additional phase transition inside the Kondo insulating state in two dimensions, which results from the appearance of the topological Kondo insulator. On the other hand, we observe that the Kondo insulating state is distinguished into three insulating phases in three dimensions, identified with the weak topological Kondo insulator, the strong topological Kondo insulator, and the normal Kondo insulator, respectively, and classified by Z topological indices. Such phase transitions inside the Kondo insulator are described by gap closing. We derive an effective Dirac theory near the gap closing momentum point for the phase transition from the strong topological Kondo insulator to the normal Kondo insulator.

## Ii Phase diagram for a topological Kondo insulator

### ii.1 An effective Anderson lattice model

In order to consider the topological nature of the heavy electron system, we construct the simplest model which introduces one -orbital and one -orbital in the unit cell. Conduction electrons are described by

 Hc=∑k∑σ(εck−μc)c†kσckσ, (1)

where is the dispersion relation of the conduction electron with momentum and real spin , and is the electron chemical potential. The -electron Hamiltonian is given by the site energy and the on-site Coulomb interaction ,

 Hf=∑i∑τ(ϵf−μc)f†iτfiτ+U∑infiτnfi−τ, (2)

where is the density operator of the -electron with pseudo-spin belonging to one representation of multiplet at site . is the time reversal partner of in the Kramers doublet. Representations other than the representation are assumed to be irrelevant.

An essential point is how these itinerant and localized electrons are coupled, generically given by the hybridization term

 Hhyb=∑k∑σ,τVστ(k)c†kσfkτ+H.c., (3)

where is the Fourier -component of . In order to determine the structure of , we have to specify the representation of -electron and its surroundings associated with conduction electron sites. If we restrict the -electron states onto the multiplet, the hopping matrix is described by

 ⟨isσ|Hhyb|jγτ⟩ ≃ ⟨isσ|Hhyb|jησ⟩⟨jησ|jmσ⟩ (4) × ⟨jmσ|jj=52μ⟩⟨jj=52μ|jγτ⟩,

where is the representation and basis of the cubic harmonic oscillator, is the -component of the orbital angular momentum of -electron, and is the -component of the total angular momentum. The approximate equality comes from the ignorance of the multiplet in estimating the matrix element.

Resorting to the table of in Ref. Takegahara_Overlap (), we can estimate s-f integrals as follows

 ⟨is|Hhyb|jxyz⟩=√15lmn(sfσ), (5) ⟨is|Hhyb|jx(5x2−3r2)⟩=12l(5l2−3)(sfσ), (6) ⟨is|Hhyb|jx(y2−z2)⟩=12√15l(m2−n2)(sfσ), (7)

where are the direction cosine of the vector . The factor of corresponds to the weight of spherical harmonics in cubic harmonics. is nothing but the Clebsch-Gordan coefficient. The last factor of of Eq. (4) is the weight of in . Using the bases in the cubic crystalline electric field for , we obtain the table of in Table I.

Next, we specify the hybridization process as the orbital of the conduction electron is situated at the same position like the case of -electron states of rare earth ions. Then, there is no local hybridization between - and -electrons. Therefore, the main hybridization process results from the nearest neighbor hopping from the -electron state at a site to the -electron state at a neighboring site , where is a vector connecting with a neighboring unit cell.

Using the hybridization matrix discussed above, we obtain the following matrix element for the hopping to the (100) direction,

 ⟨i+xs↑|Hhyb|iΓ8(1)−⟩ = ⟨i+xs↓|Hhyb|iΓ8(1)+⟩ (8) = −32√7(sfσ), ⟨i+xs↑|Hhyb|iΓ8(2)−⟩ = ⟨i+xs↓|Hhyb|iΓ8(2)+⟩ (9) = √32√7(sfσ),

for the hopping matrix element to (010) direction,

 ⟨i+ys↑|Hhyb|iΓ8(1)−⟩ = −⟨i+ys↓|Hhyb|iΓ8(1)+⟩ (10) = 32√7i(sfσ), ⟨i+ys↑|Hhyb|iΓ8(2)−⟩ = −⟨i+ys↓|Hhyb|iΓ8(2)+⟩ (11) = √32√7i(sfσ),

for the hopping matrix element to (001) direction,

 ⟨i+zs↑|Hhyb|iΓ8(2)+⟩ = −⟨i+zs↓|Hhyb|iΓ8(2)−⟩ (12) = −√37(sfσ),

respectively.

Introducing for simplicity, we obtain the hybridization Hamiltonian to (100)-direction as

 H(100)hyb = Vsf∑k([c†k↑c†k↓][02√3isinkx2√3isinkx0][fkΓ8(1)+fkΓ8(1)−] (13) +[c†k↑c†k↓][0−2isinkx−2isinkx0][fkΓ8(2)+fkΓ8(2)−])+H.c.,

for (010)-direction as

 H(010)hyb = Vsf∑k([c†k↑c†k↓][02√3sinky−2√3sinky0][fkΓ8(1)+fkΓ8(1)−] (14) +[c†k↑c†k↓][02sinky−2sinky0][fkΓ8(2)+fkΓ8(2)−])+H.c.,

for (001)-direction as

 H(001)hyb = Vsf∑k[c†k↑c†k↓][4isinkz00−4isinkz][fkΓ8(2)+fkΓ8(2)−]+H.c.. (15)

Gathering all these terms, the resulting hybridization Hamiltonian is given by

 (16)

where the hybridization matrix

 ^VΓ(k)=dΓ(k)⋅^σ (17)

is

 dΓ8(1)(k)=(2√3isinkx,2√3isinky,0), (18) dΓ8(2)(k)=(−2isinkx,2isinky,4isinkz). (19)

We note that this model Hamiltonian describes the nearest neighbor hybridization process between - and -electrons of rare earth ions in the simple cubic lattice. In appendix A we discuss how the hybridization Hamiltonian is modified when the lattice symmetry is lowered from the simple cubic to the tetragonal.

### ii.2 U(1) slave-boson mean-field analysis

We start from an effective Anderson lattice model

 H = ∑k∑σ(εck−μc)c†kσckσ+∑i∑α(ϵf−μc)f†iαfiα+ϵ∑ij∑αα′tijαα′f†iαfjα′+U2∑i∑αf†iαfiαf†i−αfi−α (20) +V∑k∑i∑ασ[dΓ(k)⋅^σσαc†kσfiαe−ik⋅ri+H.c.],

where is the hopping integral of -electrons with a parameter and the subscript in is omitted for simplicity. This effective hopping for localized fermions is introduced phenomenologically to describe possible phase transitions from the “fractionalized” Fermi liquid state Fractionalized_FL_KLM () to the Kondo insulating phase at zero temperature, which can be regarded to result from Ruderman-Kittel-Kasuya-Yoshida (RKKY) spin correlations effectively in an intermediate energy scale. Pepin_ALM_KB () The hybridization coupling term is given by Eqs. (18) and (19) for the simple cubic case. Form_Factor ()

We take the U(1) slave-boson representation for the Kondo effect in the strong coupling limit, where the localized fermion is expressed as with the single occupancy constraint . Then, the effective Lagrangian is given by

 L=∑k∑σc†kσ(∂τ−μc+εck)ckσ+V∑k∑i∑ασ[Φασ(k)c†kσb†ifiαe−ik⋅ri+H.c.] +∑i∑αf†iα(∂τ+ϵf−μc+iλi)fiα−ϵ∑ij∑αα′tijαα′f†iαb†ibjfjα′+i∑iλi(b†ibi−1), (21)

where is the Lagrange multiplier field for the constraint and .

Taking with in the saddle-point approximation, we obtain two self-consistent equations for and , given by

 λ=V2∑k∑αα′σΦασ(k)Φ∗α′σ(k)1π∫∞−∞dωnF(ω)ImGfαα′(ω+i0+,k)ω−εck+μc+i0+, (22) b2−1π∑k∑α∫∞−∞dωnF(ω)ImGfαα(ω+i0+,k)=1, (23)

respectively. is the spinon Green’s function, given by

 \boldmathGf(ω,k)=[(ω−ϵf+μc−λ)\boldmath1+ϵ\boldmatht(k)−V2b2\boldmathΦ(k)⋅\boldmathΦ∗(k)ω−εck+μc]−1, (24)

where . is the Fermi-Dirac distribution function.

### ii.3 Phase diagram

It is natural to expect a phase transition from the fractionalized Fermi liquid to the Kondo insulating state, described by the emergence of the Kondo effect () above a critical strength of hybridization. An interesting aspect of this effective lattice model is that there can exist additional phase transitions inside the Kondo insulating phase, not described by the holon condensation but characterized by the change of the Z topological index.

For simplicity we take with . Introduction of the spin dependence will not change possible phases but modify critical values of associated with their phase transitions. The dispersion relations for the heavy fermion bands are given by

 E±(k)=12(εck−ϵt(k)+ϵf+λ)−μc (25) ±12√(εck+ϵt(k)−ϵf−λ)2+4V2b2Δ2(k),

respectively, where .

In order to see how the band inversion occurs from this dispersion relation, we consider time reversal invariant momenta which satisfy , where is a reciprocal lattice vector. It is straightforward to check that there are eight time reversal invariant momentum points in three dimensions while four in two dimensions. The Z topological index, which measures how many times bands are twisted with modular two, has been reformulated for such time reversal invariant momenta in the case when the system preserves the inversion symmetry. 3D_TI_Z2index_FuKane (); 3D_TI_Z2index_Balents (); 3D_TI_Z2index_Roy () First, we observe that the band gap in the Kondo insulating phase closes linearly at some of the time reversal symmetry points which satisfy , where appears near . This identification leads us to define the parity . See appendix B. Following Dzero et al, TKI_Coleman () we can evaluate the Z topological indices given by

 ISTI = ∏mδm, (26) IαWTI = ∏mδm∣∣(k∗m)α=0, (27)

where is an index for a strong topological insulator while are indices for a weak topological insulator. We will see these indices changed, showing additional phase transitions inside the Kondo insulator.

We solve the mean field equations (22) and (23) with (24) for the simple cubic lattice with

 εck=t(k)=−2t(coskx+cosky+coskz) (28)

numerically, and take the hybridization term of the symmetry. Our slave-boson analysis uncovers 4 different phases in three dimensions, shown in Fig 1. When the hybridization coupling is smaller than a critical value V, the Kondo effect does not exist, i.e., (Phase I), giving rise to local magnetic moments decoupled from conduction electrons, where such localized spins form a spinon “Fermi” liquid state (spin liquid) due to their dispersions originating from the RKKY interaction. Fractionalized_FL_KLM () Such an exotic liquid state may be realized in geometrically frustrated lattices or at finite temperatures. Senthil_SL_nonzeroT () Increasing the hybridization coupling above V, the Kondo effect results in the formation of the heavy fermion band (), but the condition of half filling for conduction electrons leads to an insulating state instead of the heavy fermion metal. An interesting point is that the Kondo insulating state is distinguished into three insulating phases, classified by the Z topological indices. The phase II is characterized by the trivial strong topological-insulator index of , but nontrivial weak topological-insulator indices of . Thus, the phase II is identified with the weak topological Kondo insulator. The phase III is characterized by and . As a result, it is the strong topological Kondo insulator. The last Kondo insulating phase, phase IV, is the conventional Kondo insulator with trivial Z indices of and .

Since the first phase transition from the fractionalized Fermi liquid to the weak topological Kondo insulator is given by the holon condensation, i.e., the formation of the Kondo effect, it belongs to the second order transition in the saddle-point approximation. Two more phase transitions inside the Kondo insulator are described by gap closing, thus also belonging to the continuous transition. An important point is that such second order transitions inside the Kondo insulating phase are expected to be robust because singular corrections from fluctuations will not exist inside the Kondo insulator. In particular, both hybridization and gauge fluctuations will not play an important role in these phase transitions. On the other hand, such quantum corrections may be important for the nature of the first phase transition from the fractionalized Fermi liquid to the weak topological Kondo insulator, which will be discussed in the last section.

It is straightforward to check the two dimensional case, i.e., the effective Anderson lattice model on the square lattice. Indeed, we performed the same slave-boson mean-field analysis in two dimensions and found the corresponding phase diagram of Fig. 2. The main difference from the three dimensional case is the absence of the strong topological Kondo insulating phase, where the quantum “spin” Hall Kondo insulating phase exists between the fractionalized Fermi liquid and the normal Kondo insulator. Fig. 3 display the band structures of both the topological (normal) Kondo insulating phase and the critical point, where the gap closing occurs at .

### ii.4 An effective Dirac theory for the hybridization coupling with the symmetry Γ8(1)

It is given by the Dirac theory an effective field theory for the phase transition from the strong topological Kondo insulator to the normal Kondo insulator. The effective slave-boson Hamiltonian for the topological Kondo insulator is

 H = ∑kΨ†kH(k)Ψk+λ(b2−1), (29) H(k) = ⎛⎜ ⎜ ⎜ ⎜ ⎜⎝εck−μcVbΦ(k)00VbΦ∗(k)−ϵt(k)+˜ϵf−μc0000εck−μc−VbΦ∗(k)00−VbΦ(k)−ϵt(k)+˜ϵf−μc⎞⎟ ⎟ ⎟ ⎟ ⎟⎠, (30)

where is the four component Dirac spinor with , and is the form factor of the hybridization.

Expanding around the gap closing point with time reversal symmetry, , we obtain an effective Dirac theory

 H(\boldmathp)=(h(% \boldmathp)00h∗(−\boldmathp)), (31)

where is the two by two Dirac Hamiltonian with

 ε(\boldmathp) = C−D(p2x+p2y+p2z), (32) \boldmathd(\boldmathp) = (Apy,Apx,M(\boldmathp)), (33) M(\boldmathp) = M−B(p2x+p2y+p2z), (34) C = 3t+32ϵ+12˜ϵf−μc, (35) D = 12(t+ϵ2), (36) A = −2√3Vb, (37) B = 12(t−ϵ2), (38) M = 3t−32ϵ−12˜ϵf. (39)

This effective Hamiltonian turns out to be identical with that of Ref. BHZ_Original () when and are performed with . The spectrum is

 E±(\boldmathp)=ε(\boldmathp)±√A2(p2x+p2y)+M2(\boldmathp). (40)

Note that corresponds to the parity at the symmetry point . Recall that the Chern number is given by for and for in the two dimensional case. This means that the parity describes the transition from the topological insulator to the trivial insulator via gap closing.

In appendix C we derive an effective Dirac theory for the case of hybridization with .

## Iii Summary and discussion

It is essential to construct realistic lattice models in searching novel quantum states of matter. In this study we construct an effective Anderson lattice model in order to study an interplay between the topological aspect and strong correlation, regarded to be two cornerstones for novel quantum states of matter. The topological structure could be introduced in the spin-pseudospin dependent hybridization between conduction electrons and localized electrons, which originates from the interplay between the spin-orbit interaction and crystalline electric field for localized -electrons. The spin-pseudospin dependent Kondo effect plays essentially the same role as the spin-orbit coupling for topological insulators, allowing the topological Kondo insulator inside the Kondo insulating phase. Although its existence was already pointed out in an interesting recent study TKI_Coleman (), our detailed slave-boson analysis has clarified its position in the phase diagram. In addition, we can argue that the existence of the topological Kondo insulating phase is robust against quantum corrections because hybridization fluctuations cannot be strong inside the Kondo insulator and gauge fluctuations are not, either.

Unfortunately, it is difficult to say that such topological states are quite interesting because the appearance of the topological Kondo insulator just comes from the inversion of heavy fermion bands, basically the same as that in topological insulators of non-interacting electrons. However, we would like to point out that the present effective lattice model has huge potential for novel quantum states of matter. In particular, we expect interesting spin-ordering structures in the position of the fractionalized Fermi liquid state, which results from the underestimation of spin correlations in the slave-boson approach. In this respect the saddle-point analysis based on the slave-fermion theory Kim_SF () will open the possibility of fruitful spin structures, where exotic ordering of localized spins may appear as a result of the spin-dependent Kondo effect. Recently, the integer quantum Hall effect was proposed in the ferromagnetically Kondo coupled lattice model on geometrically frustrated lattices, Spin_Chirality_FKLM () where the kinetic-energy cost for conduction electrons becomes reduced due to the emergence of an internal magnetic flux from the formation of the spin chirality order. In addition, it is natural to expect the possibility of novel quantum criticality between the fractionalized Fermi liquid and the topological Kondo insulator because the band inversion occurs with the formation of the heavy-fermion band at the same time, quite uncommon in the Landau-Ginzburg-Wilson description for phase transitions. Of course, anomalous scaling near this quantum criticality is expected to be beyond the present mean-field analysis. Recently, two of us constructed an Eliashberg theory for the spin density wave transition in the surface state of the three dimensional topological insulator Kim_Takimoto_SDWQCP (), where the band reconstruction is not introduced but fluctuation corrections are incorporated. In this study we uncovered that the anomalous self-energy correction (the off diagonal self-energy in the spin or pseudospin space) is essential for self-consistency. We speculate that mathematically the same self-energy correction via hybridization fluctuations will play an important role for this nontrivial quantum criticality. In particular, we expect that the characteristic feature for this quantum critical point may be introduced in scaling of the anomalous Hall conductivity. Numerical simulations for this model seem to be invaluable.

## Acknowledgement

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MEST) (No. 2011-0074542). M.-T. was also supported by the National Foundation for Science and Technology Development (NAFOSTED) of Vietnam.

## Appendix A Model Hamiltonian in tetragonal surroundings

We consider the effect of symmetry lowering from cubic to tetragonal. In the cubic crystal structure, the level scheme of the -electron for the multiplet is given by

 D5/2↓Oh=Γ7⊕Γ8, (41)

which serves bases for the extraction of the hybridization Hamiltonian. In the tetragonal surroundings, the and bases are reduced,

 Γ7(Oh)↓D4h=Γ7, (42) Γ8(1)(Oh)↓D4h=Γ7, (43) Γ8(2)(Oh)↓D4h=Γ6, (44)

respectively. Therefore, and states can mix in the tetragonal surroundings, while doublet splits from quartet by the tetragonal crystalline electric field.

Resorting to the above discussion, we can imagine a special case that only one -electron doublet is relevant to describe the low energy physics in the tetragonal system. If the relevant doublet belongs to in the tetragonal system, the effective Hamiltonian will be

 HΓ7=Hc+Hf+HΓ7hyb, (45) Hc=∑k∑σ(εck−μc)c†kσckσ, (46) Hf=∑i∑τ(ϵf−μc)f†iτfiτ+U∑infiτnfi−τ, (47) HΓ7hyb=∑k[c†k↑c†k↓]dΓ7(k)⋅^σ[fk+fk−]+H.c., (48)

with

 εf=εfΓ7, (49) dΓ7(k)=(2√3i~Vsfsinkx,2√3i~Vsfsinky,0), (50) fkτ=fkΓ7τ, (51)

where is the relevant doublet corresponding to , and is affected from the original by the diagonalization in the tetragonal surroundings.

When the relevant doublet is , the effective Hamiltonian becomes

 HΓ6=Hc+Hf+HΓ6hyb, (52) Hc=∑k∑σ(εck−μc)c†kσckσ, (53) Hf=∑i∑τ(ϵf−μc)f†iτfiτ+U∑infiτnfi−τ, (54) HΓ6hyb=∑k[c†k↑c†k↓]dΓ6(k)⋅^σ[fk+fk−]+h.c., (55)

with

 εf=εfΓ6, (56) dΓ6(k)=(−2iVsfsinkx,2iVsfsinky,4iV′sfsinkz), (57) fkτ=fkΓ6τ, (58)

where corresponding to the hybridization to z-direction is modified by the tetragonal surroundings. Here, is the dispersion relation of conduction electrons in the tetragonal surroundings.

## Appendix B Review on the Z2 topological index

We review the parity eigenvalue in the Z topological indices of Eqs. (26) and (27). Generally speaking, any four by four matrices with the hermitian property can be decomposed by the identity matrix , five Dirac matrices , and their ten commutators . Thus, our Hamiltonian matrix can be expressed in terms of these 16 basis matrices. It is convenient to choose the following representation for the Dirac matrices,

 Γ(1,2,3,4,5)=(τz⊗σ0,τx⊗σ0,τy⊗σx,τy⊗σy,τy⊗σz),

where and are matrices in the orbital ( and ) and spin spaces, respectively.

An essential aspect is that this general expansion can be simplified near gap closing momentum points, which occur at time reversal invariant momentum points , where the gap closes linearly in momentum, thus the effective theory is given by the Dirac theory. Such a Dirac theory, referred as the Bernevig-Hughes-Zhang model, BHZ_Original () can be expanded by only and as

 HΓk∗m(k)=dΓ0(k)I+5∑a=1dΓa(k)Γa. (60)

Among these Dirac matrices, is nothing but the parity operator in the present representation,

 ^P=τz⊗σ0, (61)

where and correspond to - and - orbitals, respectively. It should be noted the following relations on parity

 ^PΓa^P−1=Γafora=1, (62) ^PΓa^P−1=−Γafora≠1. (63)

Therefore, is only parity-even Dirac operator.

For the three dimensional system, there are eight time reversal invariant points in the first Brillouin zone. We obtain two invariants at such time reversal invariant points, given by

 ΘHΓ(k∗m)Θ−1=HΓ(k∗m), (64) ^PHΓ(k∗m)^P−1=HΓ(k∗m), (65)

where is the time reversal operator. Considering the relation of Dirac matrices on parity transformation, we can easily estimate the eigenvalue at

 EΓ(k∗m)=dΓ0(k∗m)I+dΓ1(k∗m)^P, (66)

where is given by

 dΓ1(k∗m)=12(εck∗m+ϵt(k∗m)−ϵf−λ). (67)

Then, the parity eigenvalue at can expressed as follows

 δm=−sgn(dΓ1(k∗m))=−sgn(εck∗m+ϵt(k∗m)−ϵf−λ).

## Appendix C An effective Dirac theory for the hybridization coupling with the symmetry Γ8(2)

In the hybridization with the effective Hamiltonian is

 H(k)=⎛⎜ ⎜ ⎜ ⎜ ⎜⎝εck−μcVb(−2isinkx+2sinky)04VbisinkzVb(2isinkx+2sinky)−ϵt(k)+˜ϵf−μc4Vbisinkz00−4Vbisinkzεck−μc−Vb(2isinkx+2sinky)−4Vbisinkz0−Vb(−2isinkx+2sinky)−ϵt(k)+˜ϵf−μc⎞⎟ ⎟ ⎟ ⎟ ⎟⎠. (69)

Expanding this Hamiltonian around the point, we reach the following expression for the Dirac theory

 H(\boldmathp)=(h(p)g(