# Spin Polarization of Gapped Dirac Surface States Near the Topological Phase Transition in TlBi(SSe)

## Abstract

We performed systematic spin- and angle-resolved photoemission spectroscopy of TlBi(SSe) which undergoes a topological phase transition at 0.5. In TlBiSe ( = 1.0), we revealed a helical spin texture of Dirac-cone surface states with an intrinsic in-plane spin polarization of 0.8. The spin polarization still survives in the gapped surface states at 0.5, although it gradually weakens upon approaching = 0.5 and vanishes in the non-topological phase. No evidence for the out-of-plane spin polarization was found irrespective of and momentum. The present results unambiguously indicate the topological origin of the gapped Dirac surface states, and also impose a constraint on models to explain the origin of mass acquisition of Dirac fermions.

###### pacs:

73.20.-r, 71.20.-b, 75.70.Tj, 79.60.-iThree-dimensional topological insulators (TIs) exhibit a novel quantum state with metallic topological surface state (SS) which disperses across the bulk band gap generated by a strong spin-orbit coupling. Specifically, the topological SS is characterized by a linearly dispersing Dirac-cone energy band with the helical spin texture (1); (2); (3); (4), which hosts massless Dirac fermions protected by time-reveal symmetry (TRS). This peculiar SS of TIs provides a platform for fascinating quantum phenomena such as the robustness against nonmagnetic impurities / disorder (5); (6) and the emergence of Majorana fermions (7). Breaking the TRS by a magnetic field or a magnetic order lifts the band degeneracy at the Dirac point and causes an energy gap called a Dirac gap in the topological SS, turning the massless Dirac fermions into a massive state. Theoretically, realization of the massive state is a prerequisite to novel topological phenomena such as the topological magnetoelectric effect and half-integer quantum Hall effect (8); (9), but experimentally, the role of TRS-breaking magnetic impurities in triggering the Dirac-gap opening in the topological SS is under intense debate (10); (11); (12); (13).

Recently, it was found by angle-resolved photoemission spectroscopy (ARPES) measurements that the thallium-based ternary chalcogenide solid-solution TlBi(SSe) goes through a quantum phase transition (QPT) from the topological ( 0.5) to the non-topological ( 0.5) phase (14); (15). Intriguingly, the Dirac-cone topological SS in TlBiSe ( = 1.0) starts to show a Dirac gap upon partially replacing Se with S, and the gap gradually grows upon increasing the S content until the QPT at = 0.5 eliminates the SS (15). Since there is no evidence for magnetic impurities in the measured crystals, it has been puzzling how the Dirac fermions in TlBi(SSe) acquire the mass without explicitly breaking the TRS. To gain insights into the microscopic mechanism of this intriguing mass acquisition, it is of particular importance to establish the spin structure of the gapped SS in TlBi(SSe), since the topological or non-topological nature of the SS as well as the origin of the Dirac gap is intimately related to the characteristics of the spin texture.

In this Letter, we report our spin-resolved ARPES results of TlBi(SSe) with various values ( = 1.0, 0.9, 0.8, 0.6, and 0.0), and present the evolution of the spin texture as a function of . We demonstrate that the helical spin texture persists even in the gapped SS observed at 0.5, while the spin polarization vanishes in the ARPES spectrum in the non-topological phase at 0.5 in accordance with the absence of the SS. In addition, we found unusual reduction of the spin polarization in the SS near the QPT. We discuss the implications of the present experimental findings on the mechanism of the Dirac gap opening.

High-quality single crystals of TlBi(SSe) were grown by a modified Bridgman method. Details of the sample preparations were described elsewhere (15). ARPES measurements were performed with a spin-resolved photoemission spectrometer based on MBS-A1 analyzer at Tohoku University (16). We used one of the Xe I lines ( = 8.437 eV) to excite photoelectrons. Samples were cleaved along the (111) crystal plane in an ultrahigh vacuum of 510 Torr. The energy resolution for the spin-resolved and regular ARPES measurements was set at 40 and 6 meV, respectively. The sample temperature was kept at 30 K during the measurements. We used the Sherman function value of 0.07 to obtain spin-resolved ARPES data (16).

First we demonstrate the spin structure of TlBiSe ( = 1.0) in which the topological origin of the Dirac-cone SS is well established by previous studies (17); (18); (19). Figure 1(a) shows the ARPES intensity plot at of TlBiSe around the point as a function of in-plane wave vector and . We clearly observe circular Fermi surface (FS) with hexagonal deformation, which originates from the upper branch of the Dirac-cone SS as displayed in Fig. 1(b). We also find a finite intensity distribution around the point as indicated by white arrow in Fig. 1(b). This signal is attributed to the bulk conduction band (CB) which is slightly occupied due to electron doping caused by Se vacancies in the crystals (17); (18); (19).

Figures 1(c) and (d) show spin-resolved energy distribution curves (EDCs) of TlBiSe for the in-plane spin component measured at various k points along the cut as indicated in Fig. 1(a). We define the spin polarization vector perpendicular to the measured k [see, , thick red arrow in Fig. 1(a)], where the “up spin” points to the clockwise direction. One can immediately recognize that the up-spin EDC is always more prominent than the down-spin counterpart in all the measured k region, consistent with the spin helical texture of the Dirac-cone SS (4); (14). Upon closer look, one finds that the shape of the up-spin EDC depends on the k point. For instance, near the point (point E and J) the peak in the up-spin EDC (marked by circle) is located at 0.3 eV, but it approaches on moving away from the point (from E to A, or from J to F), reflecting an electronlike dispersion of the upper branch of the Dirac cone (note that the dispersive feature is less clear than the regular ARPES data because of the wider k window and the lower energy resolution of the spin-resolved mode). On the other hand, the peak in the down-spin EDC is always pinned at . This is because it is governed by the bulk CB which contributes equally to the up- and down-spin EDCs around the point (as evidenced by the small difference in the near- spectral weight between the up- and down-spin data at points D, E, I, and J) as a result of the spin-degenerated character.

Figure 2(a) shows the near- ARPES intensity plot of TlBi(SSe) measured along the direction () for various values ( = 1.0-0.6 in the topological phase and = 0.0 in the non-topological phase). The “X”-shaped topological SS transforms into the gapped SS in = 0.9-0.6, as one can see from the suppression of the ARPES intensity at the Dirac point and the parabolic-like dispersion of the upper branch of the Dirac cone. The characteristics of the gapped SS and its intrinsic nature are demonstrated in Supplemental Materials, where it is explicitly shown that the Dirac gap opening in our data is not due to a spectral-weight suppression at the Dirac point, momentum broadening, bad cleaves, nor a random disorder effect. At = 0.0 (TlBiS), the SS completely disappears and only the bulk CB is observed at , reflecting the non-topological nature (15).

To elucidate the evolution of the spin structure of the SS, we have performed spin-resolved ARPES measurements by focusing on a particular k point. The k point was chosen to be located midway between the Fermi vector () and the point along the direction [see vertical red (thick) line in Fig. 2(a)] so that the binding energy () of the surface band ( 0.25 eV) is apart from that of the bulk CB. As can be seen in Fig. 2(b), the shape of the spin-resolved EDCs for the in-plane component is similar between = 1.0 and 0.9. In addition, the spin polarization () [Fig. 2(c)] exhibits a similar value of 0.3, suggesting that the gapped SS in = 0.9 has a helical spin texture as in = 1.0. Another notable aspect is that the in-plane spin polarization is gradually reduced upon approaching the QPT at 0.5, as evidenced by the smaller difference between the up- and down-spin EDCs for the samples with lower values. The value in = 0.6 is 0.15 (half of that at = 1.0), and it vanishes in the non-topological phase ( = 0.0).

To discuss more quantitatively the absolute spin polarization of the SS, it is necessary to remove the finite contribution from the spin-degenerated bulk band. For this purpose, we have fit the EDCs in the regular ARPES measurement by incorporating both the surface and bulk components [see Fig. 3(a)], and estimated the spectral weight of the SS () with respect to that of the total weight ( = + ). The intrinsic in-plane spin polarization of the SS () corresponds to the value divided by /. (For the detailed procedure to estimate the spin polarization and corresponding error bars, see Supplemental Materials). Figures 3(b) and (c) display the -dependence of / and ( is also plotted for reference), respectively. As can be seen in Fig. 3(c), the estimated value, which reaches 0.8 for = 1.0, is gradually reduced with increasing to become 0.5 at = 0.6, and finally vanishes in the non-topological phase at = 0.0. The origin of the reduction in the in-plane spin polarization toward the QPT in the gapped SS is not clear, but one can infer that it is not due to contamination from bulk conduction bands or the degenerate surface two-dimensional electron gas (2DEG) (20), since the effect of the former is already discounted in our analysis and the latter is obviously not formed in our surface situation. The less-than-unity value of at = 1.0 is likely to be a consequence of the spin-orbit entanglement as proposed from first-principle calculations (21).

The next important issue to clarify is the out-of-plane () spin component. Figure 4(a) displays the spin-resolved EDCs and corresponding spin polarizations for the -spin component measured at the same k point as in Fig. 2 (along the direction) for the samples in the topological phase ( = 1.0, 0.8, and 0.6). Spin-resolved EDCs for the k point along the direction are also displayed for = 1.0; obviously, there is no apparent -spin component in the nor the directions. It was previously found (4); (22); (23) that in BiTe a finite -spin texture emerges along the direction due to strong hexagonal warping of the SS. However, the hexagonal warping effect is much weaker in TlBi(SSe) than in BiTe, resulting in no or negligible -spin component (4). The present data reconfirms this previous result for the whole topological phase, where the estimated -axis spin polarization remains negligible (-0.02 +0.01).

While the above result demonstrates no -spin polarization near the point, a natural question arises as to whether any -spin component exists exactly at the point. To address this issue, first we carefully examined the EDCs for = 0.9 [Fig. 4(b)] and determined the spectrum which exactly corresponds to the point (thick line), and then performed the spin-resolved ARPES measurement at this k point with a higher energy resolution (15 meV). As shown in Fig. 4(c), we have succeeded in resolving two well-separated peaks in the spin-resolved EDCs attributed to the upper and lower branches of the gapped SS, respectively. Notably, there is essentially no difference between the up- and down-spin EDCs, indicating null -spin polarization. These results suggest that the -spin component is absent irrespective of and k. Based on these observations, we depict in Fig. 4(d) the schematic picture of the spin-texture evolution for three different phases (massless Dirac, massive Dirac, and non-topological phases) in TlBi(SSe).

Now we discuss the implications of the observed characteristics of the spin polarization in relation to the Dirac-gap opening. First, the present result directly addresses the question whether the gapped SS has a topological origin. One may argue that the topological phase transition should have occurred between the massless ( = 1.0) and the massive ( = 0.9) phase, and that the observed gapped SS corresponds simply to a pair of trivial, spin-degenerate SSs which happened to present a gap-like feature at the point. In this respect, such a possibility is immediately ruled out by the observation of the spin non-degenerate nature of the gapped SS.

Another possible origin of the gapped SS is the hybridization of two topological SSs. In ultrathin films of BiSe, it was reported that a Dirac gap opens when the electron wavefunctions of the top and bottom surfaces start to hybridize (24); (25). In this case, the hybridized state is spin degenerate and loses the original helical polarization (26). While the thick nature of our samples ( 100 m) already spoke against this possibility (15), the spin non-degeneracy observed here conclusively rules out the hybridization scenario. Moreover, a recent theory for ultrathin films in the hybridized state predicted a finite -spin component (25), which was not observed in the present experiment. It is worth noting that recent ARPES studies on BiSe ultrathin films (27) and bulk heterostructures involving BiSe units (28) have reported a second Dirac cone which originates from the backside surface (interface). If the hybridization of the wave function between the Dirac cone at the topmost surface and the second Dirac cone at a backside surface (which may accidentally form due to bad cleaves) were the source of the observed phenomena, the separation between the topmost surface and the accidentally-formed backside surface must become systematically closer with decreasing , which is extremely unlikely.

An important question is whether the TRS is broken in the massive Dirac phase. When the topological SS feels a magnetic field or magnetization directed perpendicular to the surface (along the -axis), it obtains a Dirac gap and the spin vector at the point is aligned along the -axis in opposite directions in the lower and upper branches of the gapped SS (9). In this respect, our finding that there is no -spin component in the gapped SS does not give any support to this scenario. Nevertheless, one should keep in mind that if there are magnetic domains with antiparallel polarizations, the overall -spin component can be canceled. Hence, it is difficult to draw any firm conclusion about the TRS breaking from the present experiment.

A viable possibility for the origin of the Dirac gap is the bulk-surface coupling. A recent theory suggested that the Dirac-cone feature can be destroyed by strong potential impurities which lead to bulk-assisted scattering to allow virtual spin-flip excitations (29). The proximity of the bulk band to the Dirac point near the QPT (15) and the disorder due to random replacement of S with Se may be in line with this scenario. Another possibility could be critical fluctuations associated with the QPT as mentioned in Ref. (15), and the reduction of the net spin polarization toward the QPT may be in line with this scenario; however, no evidence for such critical fluctuations has been obtained so far. While a comprehensive answer to the origin of the Dirac gap has not yet been obtained, the present observation that established the nature of the spin texture and its evolution in the gapped phase provides an important step toward microscopic understanding of the topological QPT and the unconventional mass acquisition of Dirac fermions in TIs.

In summary, we have reported our spin-resolved ARPES experiments on TlBi(SSe) to determine the spin polarization as a function of . Besides firming up the intrinsic nature of the Dirac gap observed for 0.6 0.9, we found that the helical spin texture survives in the massive Dirac phase, which clearly points to the topological nature of the gapped SS. While the exact mechanism of the Dirac gap opening is yet to be elucidated, the direct connection between the emergence of the helical spin texture and the topological QPT, as well as the reduction of the net spin polarization toward to the QPT, gives an important clue.

###### Acknowledgements.

We thank K. Kosaka and T. Arakane for their assistance in ARPES measurements, and T. Minami for his help in sample preparations. This work was supported by JSPS (NEXT Program and KAKENHI 23224010), JST-CREST, MEXT of Japan (Innovative Area “Topological Quantum Phenomena”), AFOSR (AOARD 104103 and 124038), and KEK-PF (Proposal number: 2012S2-001).Supplemental Materials for “Spin Polarization of Gapped Dirac Surface States Near the Topological Phase Transition of TlBi(SSe)”

S. Souma, M. Komastsu, M. Nomura, T. Sato, A. Takayama, T. Takahashi,

K. Eto, Kouji Segawa, and Yoichi Ando

WPI Research Center, Advanced Institute for Materials Research, Tohoku University, Sendai 980-8577, Japan

Department of Physics, Tohoku University, Sendai 980-8578, Japan

Institute of Scientific and Industrial Research, Osaka University, Ibaraki, Osaka 567-0047, Japan

## .1 S1. Characteristics of the Dirac gap

To elucidate the characteristics of the gapped surface state (SS) in more detail, we show in Fig. S1 the energy distribution curves (EDCs) at the point for TlBi(SSe) at each selenium composition in the topological phase. As one can see in this plot, the EDC at the point for = 1.0 is composed of a single peak at the Dirac point (DP) which is satisfactorily fitted by a single Lorentzian peak. On the other hand, we observe two well-separated peaks for = 0.9-0.6 (though the peaks for = 0.6 are broad), which are very well reproduced by two Lorentzian peaks. If the energy gap of the SS (the Dirac gap) were caused by a spectral-weight suppression at the DP as proposed in Ref.(30), there is no reason that the experimental data presenting a two-peak feature (particularly the data points between the two peaks) are fitted well by a pair of Lorentzians. Our result thus suggests that the observed two-peak structure reflects an intrinsic spectral function of the SS with an energy gap at the DP.

We note that the data for the gapped phase displayed in Fig. S1 were measured independently from our previous published work (15) using different samples, and yet, the Dirac gap shows a quantitatively similar size indicative of its reproducibility. Specifically, the gap size in our previous experiment (15), as estimated from the peak position of the second-derivative of the EDCs, were meV for = 0.9, 0.8, and 0.6, respectively, and those in the present study as estimated from the energy separation of the peak position of two Lorentzians are meV, respectively, which agree well within the experimental uncertainties. In fact, as shown in Fig. S2 which plots the cleaving dependence of the band dispersion for = 0.9, the obtained gap feature is robust: the gap always exhibits the same value ( meV) despite the finite variations in the overall intensity distribution for different cleaves. This result is obviously incompatible with the recent proposal (30) that the “gap” can be caused by bad cleaves and rough surfaces, which would lead to sizable variation in the gap value for different cleaving conditions. It should also be noted that if the gap is caused by a momentum broadening, the spectral linewidth for the gapped sample should be broader than that for the samples showing no gap. However, as demonstrated in Fig. S3, the peak width of the momentum distribution curves (MDCs) for = 0.9 is comparable to or even narrower than that for = 1.0, indicating that the momentum-broadening effect is unlikely to be the source of the gap opening.

One may argue that a “gap” could be caused by strong disorder and resultant smearing of the spectral weight at the DP. If this is the case, it is naturally expected that a similar gap appears in other solid-solution systems of topological insulators. In this respect, it is useful to compare the present result with BiSbTeSe (BSTS) (31) and Pb(BiSb)Te (Pb124) (32) where similar or even stronger random disorder is introduced. As one can see in Fig. S4, we found no evidence for a Dirac gap in both BSTS and Pb124 systems regardless of the ordered or disordered compositions, since the upper and lower branches of the Dirac cone are obviously degenerate exactly at the point. Hence, in our experiments the Dirac gap has so far been only observed in the TlBi(SSe) system in proximity to a topological QPT.

## .2 S2. Procedure to estimate the spin polarization and its accuracy

In the analysis of the spin-resolved ARPES data, we first obtained the scattering asymmetry parameter at each binding energy where and are the number of electrons detected by the left-hand-side and right-hand-side channeltrons in the Mott spin polarimeter, respectively. Next we obtained the in-plane spin polarization through the formula where is the effective Sherman function ( = 0.07; Ref.(16)). The statistical error in () is directly linked to that in through . Since , we accumulated all the spectra in Figs. 2(b) and (c) until the same error-bar range is achieved for the value when it is averaged over the energy region of 0.1 eV centered at the surface-band energy. We plot this averaged value as a dashed line in Fig. 2(c) (with its actual value indicated), and also by triangles (with error bars of ) in Fig. 3(c). The error bars in the intrinsic spin polarization [] were evaluated from that in and by taking into account the error propagation.

## .3 References

[15] T. Sato et al., Nature Phys. 7, 840 (2011).

[16] S. Souma et al., Rev. Sci. Instrum. 81, 095101 (2010).

[30] S.-Y. Xu et al., arXiv:1206.0278.

[31] T. Arakane et al., Nature Commun. 3, 636 (2012).

[32] S. Souma et al., Phys. Rev. Lett. 108, 116801 (2012).

### References

- M. Z. Hasan and C. L. Kane, Rev. Mod. Phys. 82, 3045 (2010).
- X.-L. Qi and S.-C. Zhang, Rev. Mod. Phys. 83, 1057 (2011).
- Y. Xia et al., Nature Phys. 5, 398 (2009).
- S. Souma et al., Phys. Rev. Lett. 106, 216803 (2011).
- A. A. Taskin, S. Sasaki, K. Segawa, and Y. Ando, Phys. Rev. Lett. 109, 066803 (2012).
- J. Seo et al., Nature (London) 463, 343 (2010).
- L. Fu and C. L. Kane, Phys. Rev. Lett. 100, 096407 (2008).
- X.-L. Qi et al., Science 323, 1184 (2009).
- X.-L. Qi, T. L. Hughes, and S.-C. Zhang, Phys. Rev. B 78, 195424 (2008).
- Y. L. Chen et al., Science 329, 659 (2010).
- T. Valla, Z. H. Pan, D. Gardner, Y. S. Lee, and S. Chu, Phys. Rev. Lett. 108, 117601 (2012).
- M. R. Scholz et al., Phys. Rev. Lett. 108, 256810 (2012).
- L. A. Wray et al., Nature Phys. 7, 32 (2010).
- S.-Y. Xu et al., Science 332, 560 (2011).
- T. Sato et al., Nature Phys. 7, 840 (2011).
- S. Souma et al., Rev. Sci. Instrum. 81, 095101 (2010).
- T. Sato et al., Phys. Rev. Lett. 105, 136802 (2010).
- Y. L. Chen et al., Phys. Rev. Lett. 105, 266401 (2010).
- K. Kuroda et al., Phys. Rev. Lett. 105, 146801 (2010).
- M. Bianchi et al., Nature Commun. 1, 128 (2010).
- O. V. Yazyev, J. E. Moore, and S. G. Louie, Phys. Rev. Lett. 105, 266806 (2010).
- Y. H. Wang et al., Phys. Rev. Lett. 107, 207602 (2011).
- S.-Y. Xu et al., arXiv:1101.3985.
- Yi Zhang et al., Nature Phys. 6, 584 (2010).
- H.-Z. Lu, W. Y. Shan, W. Yao, Q. Niu, and S. Q. Shen, Phys. Rev. B 81, 115407 (2010).
- A. Takayama et al., Nano Lett. 12, 1776 (2012).
- M. H. Berntsen, O. Götberg, and O. Tjernberg, arXiv:1206.4183.
- K. Nakayama et al., arXiv:1206.7043.
- A. M. Black-Schaffer and A. V. Balatsky, Phys. Rev. B 85, 121103(R) (2012).
- S.-Y. Xu et al., arXiv:1206.0278.
- T. Arakane et al., Nature Commun. 3, 636 (2012).
- S. Souma et al., Phys. Rev. Lett. 108, 116801 (2012).