# Exchange and collective behavior of magnetic impurities in a disordered helical metal

## Abstract

We study the exchange interaction and the subsequent collective behavior of magnetic impurities embedded in a disordered two-dimensional (2D) helical metal. The exchange coupling follows a statistical distribution whose moments are calculated to the lowest order in , where is the Fermi momentum of itinerant electrons and is the mean free path. We find that i) the first moment of the distribution decays exponentially, and ii) the variance of the interaction is long-range, however, it becomes independent of the orientation of the localized magnetic moments due to the locking between spin and momentum of the electrons that mediate the interaction. As consequence, long-range magnetic order tends to be suppressed, and a spin glass phase emerges. The formalism is applied to the surface states of a three-dimensional (3D) topological insulator. The lack of a net magnetic moment in the glassy phase and the full randomization of spin polarization at distances larger than excludes a spectral gap for surface states. Hence, non-magnetic disorder may explain the dispersion in results for photoemission experiments in magnetically-doped topological insulators.

###### pacs:

73.20.-r,75.30.Hx,75.50.Lk,75.70.TjA topological insulator is a system that supports metallic edge/surface states within the bulk gap, whose existence and integrity are protected by time-reversal symmetry (1). In the case of a 3D topological insulator, the surface states disperse as Dirac quasi-particles; the minimal Hamiltonian consists in a Bychkov-Rashba spin-orbit coupling of the form ()

(1) |

where is the Fermi velocity of surface electrons and is a vector of Pauli matrices acting on electron spin. The inclusion of a mass term in the Hamiltonian, , reflects the breakdown of time-reversal symmetry. This may arise as a Zeeman term due to a weak magnetic field (2) or the proximity exchange coupling to a magnetic thin film (3). In this scenario, the system becomes a 2D quantum Hall liquid, a perfect platform for several magneto-electric effects (7).

An interesting possibility is the deposition of magnetic adatoms, in such a way that, for high enough concentrations, , and at low enough temperatures, , a spontaneous ordering opens a gap in the spectrum, being the mass proportional to the net magnetic moment in the out-of-plane direction (4); (5); (6). Experimentally, this possibility does not seem to be completely clear. Angle-resolved photoemission (ARPES) and scanning tunnel spectroscopy (STS), together with X-ray magnetic circular dichroism (XMCD) and magneto-transport experiments generate results that are in agreement with a gap opening in some cases (8) and gapless spectrum in others (9) for similar experimental conditions. In the particular case of Fe adatoms deposited directly on the surface of BiSe, for example, a gap of the order of 100 meV at K was reported in an ARPES experiment (10), whereas later on, another ARPES experiment in similar conditions of Fe concentration and even lower temperatures reported no gap opening (11), further confirmed by STS and XMCD experiments (12). This dispersion in experimental results motivates the present study about the impact of disorder in the collective behavior of magnetic adatoms placed on the surface of a 3D topological insulator.

We consider first the problem of the exchange interaction between localized spins associated to magnetic impurities embedded in a weakly disordered helical metal. The Hamiltonian reads in general

where the first terms refers only to the itinerant electrons and the second term describes the coupling with localized spins,

where the sum runs over the magnetic impurities. Here , are the spin operators of the impurity and , are the spin density operators of the metal evaluated at the magnetic impurity sites,

The exchange is taken to be anisotropic due to the 2D nature and strong spin-orbit coupling of the system. In-plane isotropy is assumed, .

The single-particle Hamiltonian for itinerant electrons can be written as , where the first term corresponds to Eq. (1), and the second describes the effect of disorder, . Here is the potential created by a non-magnetic impurity or any source of disorder that preserves time-reversal symmetry. Since we do not posses a detailed expression for and the distribution of impurities change from sample to sample, we employ a statistical description in terms of different disorder realizations forming an ensemble of macroscopically identical replicas of the system. Assuming that the typical decay length of the impurity potential is smaller than the mean separation between scattering centers, we consider a Gaussian distribution of disorder configurations, determined by the mean free path , or equivalently, the scattering time , and characterized by correlators of the form

where is the density of states at the Fermi level, . The itinerant electrons mediate the Ruderman-Kittel-Kasuya-Yosida (RKKY) interaction (13) between the localized magnetic moments. For a dilute concentration of magnetic impurities this can be studied within second order perturbation theory. Then, the effective exchange coupling is determined by the static spin susceptibility of the helical electron gas. In the presence of non-magnetic disorder, this follows a statistical distribution whose moments can be computed by standard diagrammatic techniques. The first moment of the distribution is defined as , where we emphasize with the notation that translation invariance is restored on average. To the lowest order in we can write

where is the disorder-averaged Green operator in the Matsubara frequencies domain, and the trace is taken in the spin indices. Here we are neglecting vertex corrections, which is strictly valid at , whereas the self-energy corrections are introduced by means of in the self-consistent Born approximation. The hierarchy of length scales of the problem for which this equation strictly applies is , as illustrated in Fig. 1. This corresponds to a classical diffusive regime: helical electrons connecting a pair of localized spins describe a classical diffusion path suffering several collisions with non-magnetic scatterers. In the limit, the disorder-averaged RKKY interaction reads

(2) |

controls the exchange between the out-of-plane spin components, and between the projections along the direction linking the two magnetic impurities. The last term is a Dzyaloshinskii-Moriya coupling. In the diffusive regime, the three components decay exponentially as in a conventional 2D electron gas (14). This must not be interpreted as an exponential suppression of the RKKY interaction, but as the result of the randomization of its characteristic oscillatory tail (15). Consequently, higher moments of the distribution must be studied.

The calculation of the second moment of the distribution, , is analogous to the calculation of the universal conductance fluctuations in a disordered conductor (16). To the lowest order in , the second moment is given by both diffuson and cooperon ladder contributions, see Fig. 2. In the absence of time-reversal symmetry breaking perturbations (excluding the localized spins) both contributions are equal. We can write

where the diffuson modes are introduced as

Here corresponds to the identity and the contraction in spin indices (latin labels) is assumed. Diagrammatically, corresponds to the boxes in the first diagram of Fig. 2. At this point, it is useful to introduce the matrix

In the diffusive regime, , with and (otherwise, the diffuson ladder is zero), the diffuson satisfy the equation

(3) |

where is the diffusion constant. The triplet modes () are coupled by precession terms and only the singlet mode remains gapless . Therefore, the triplet modes yields to an exponentially vanishing contribution to , similarly to what happens in a 2D electron gas with strong spin-orbit scattering (17). Only the singlet mode contribution survives, leading in the limit to

(4) |

As it is evident from this result, the RKKY coupling is still a long-range interaction, however, the variance of the interaction becomes independent of the orientation of the localized spins, .

Different magnetically ordered phases have been proposed in the literature (4); (5); (6). In diffusive media, however, these phases are expected to be suppressed due to the exponential decay of the first moment of the distribution for the exchange coupling, Eq. (2), since, according to mean field arguments, it is this quantity what determines the critical temperature below which long-range magnetic order is possible. On the other hand, disorder introduces non-negligible fluctuations of the exchange coupling. In the particular case of an helical metal, the locking between momentum and spin of carriers makes the spin response independent of the orientation of the localized spins, Eq. (4). Due to these fluctuations, we expect that localized spins freeze to a non-zero value at low temperatures, but with no spatial correlation between them and zero net magnetization. In such a glassy phase, a global spectral gap is excluded.

We focus on the collective behavior of magnetic adatoms assuming that their spatial distribution is completely random. The clean limit with the Fermi level lying at the Dirac point was analyzed in Ref. (6). The form of the interaction in that case makes the in-plane interactions frustrated. Then, different behaviors are expected as a function of . For , a Ising-like ferromagnetic phase is expected at low temperatures, with magnetization along the out-of-plane axis. Hence, a gap in the spectrum of surface states is expected. In the opposite limit, , the frustrated in-plane interactions dominate, giving rise to a spin glass separated from the Ising ferromagnet by a quantum critical point estimated to be located at (6). Similarly, fluctuations in the exchange interaction due to disorder tend to destroy long-range order, and the system freezes to a glass for sufficiently low temperatures.

We analyze in detail the situation when . The spin Hamiltonian reads

where the couplings follow a certain statistical distribution. The first and second moments can be written in general as

Here describes the statistical distribution of magnetic adatoms over the surface of the topological insulator. We assume for simplicity that they are uniformly distributed, so then we can approximate the above equations by (18)

(5) |

and similarly for the second moment,

(6) |

Here is the area of the system and , the number of localized spins. Note that the integral in Eq. (6) is infrared divergent, but the result of Eq. (4) is strictly valid at ; therefore, there is a natural cut-off for this integral determined by the mean free path, which allows to estimate to the leading order in .

We assume that higher moments of the distribution do not affect the thermodynamic properties of the system, which can be studied within the Sherrington-Kirkpatrick (SK) model (19). Following SK, the saddle-point equations reads

(7) | |||

(8) |

where is the magnetization per spin and is the Edwards-Anderson order parameter (20), and we have introduced

Here we have taken , where represents the concentration of non-magnetic strong scatterers.

For low concentration of scatterers the model predicts a second order phase transition to a ferromagnetic state at . In the opposite limit, fluctuations in the couplings destroy long-range order, and a spin glass phase characterized by but is stabilized below . The phase diagram is shown in Fig. 3. Numerical solution of Eqs. (7)-(8) predicts a region of the parameter space where the system passes from a paramagnetic to a ferromagnetic to a reentrant spin glass phase (dashed line in Fig. 3) as the temperature is reduced. This is strange since at intermediate temperatures, where entropy plays a role, the system is ordered, but as one finds that a disordered phase is preferred. This is associated to the instability of the saddle-point solution of the SK model and the replica symmetry breaking. From Parisi’s solution (21) we predict a transition from the spin glass to a modified ferromagnetic phase (sometimes called mixed phase, where the replica symmetry is broken) at . The boundary between this modified ferromagnetic phase and a conventional Ising ferromagnet can be estimated from the Almeida-Thouless line (22).

Within this mean-field scenario, the formation of a spin glass at temperatures well below can be seen as quenched fluctuations of a mass term in Eq. (1). Net magnetization is absent beyond the disorder threshold , and the situation resembles the quantum percolation picture of the plateau transition of the quantum Hall effect (23), where the mass plays the role of the control parameter. These fluctuations are marginally irrelevant in the renormalization group sense, whereas higher moments are marginal at the tree level, what justifies our previous assumptions.

In summary, we have computed the first moments of the distribution for the RKKY interaction in a weakly disorder helical metal and applied the results to infer some trends in the collective behavior of magnetic adatoms on a 3D topological insulator. For , our analysis reveals that the ensemble of localized spins freezes to a spin glass phase above the disorder threshold . In-plane exchange interactions may help to stabilize this phase for weaker disorder due to frustration. The randomization in the orientation of the spins of the magnetic adatoms due to the helicity of carriers enforces this picture. In the spin glass phase, a spectral gap is precluded due to the lack of a net magnetic moment. This means that an ARPES experiment would report no gap opening in this scenario. Therefore, the presence of disorder may explain the dispersion in results for photoemission experiments in magnetically-doped topological insulators. Nevertheless, a local probe (24) could test the consequences of the breakdown of time reversal symmetry due to the freezing of the moments of the magnetic adatoms.

Finally, it is worth to mention that the effect of a time-reversal symmetry breaking in the electron gas that mediates the interaction is not taken into account in a self-consistent way. In a recent calculation, the authors of Ref. (25) showed that a gap opening does not change the general structure of the RKKY interaction. Similarly, effects of dissipation can be neglected at first glance based on the argument that the RKKY interaction depends on electronic states deep inside the Fermi sea, not only at the Fermi surface, whereas dissipative phenomena as the Kondo effect is purely a Fermi-surface effect (26). In other words, a rearrangement of the low energy part of the spectrum does not dramatically affect the RKKY interaction. Nevertheless, interesting novel physics beyond the scope of this work may appear at low temperatures, particularly in the disordered phase.

The author acknowledges support from the European Union’s Seventh Framework Programme (FP7/2007-2013) through the ERC Advanced Grant NOVGRAPHENE (GA No. 290846), and useful discussions with A. Asenjo-Garcia, A. Cortijo and M. A. Cazalilla during the preparation of the manuscript.

### References

- M. Z. Hasan and C. L. Kane, Rev. Mod. Phys. 82, 3045 (2010); Xiao-Liang Qi and Shou-Cheng Zhang, ibid 83, 1057 (2011).
- Dong-Xia Qu, Y. S. Hor, Jun Xiong, R. J. Cava, and N. P. Ong, Science 329, 821 (2010).
- Flavio S. Nogueira and Ilya Eremin, Phys. Rev. Lett. 109, 237203 (2012); László Oroszlány and Alberto Cortijo, Phys. Rev. B 86, 195427 (2012); S. V. Eremeev, V. N. Men’shov, V. V. Tugushev, P. M. Echenique, and E. V. Chulkov, ibid. 88, 144430 (2013).
- Qin Liu, Chao-Xing Liu, Cenke Xu, Xiao-Liang Qi, and Shou-Cheng Zhang, Phys. Rev. Lett. 102, 156603 (2009).
- Jia-Ji Zhu, Dao-Xin Yao, Shou-Cheng Zhang, and Kai Chang, Phys. Rev. Lett. 106, 097201 (2011).
- D. A. Abanin and D. A. Pesin, Phys. Rev. Lett. 106, 136802 (2011).
- Xiao-Liang Qi, Taylor L. Hughes, and Shou-Cheng Zhang, Phys. Rev. B 78, 195424 (2008); Andrew M. Essin, Joel E. Moore, and David Vanderbilt, Phys. Rev. Lett. 102, 146805 (2009); Ion Garate and M. Franz, ibid. 104, 146802 (2010).
- Y. L. Chen et al., Science 329, 659 (2010); Y. S. Hor et al., Phys. Rev. B 81 195203 (2010); Su-Yang Xu et al., Nat. Phys. 8, 616 (2012); Joseph G. Checkelsky et al., ibid., 729 (2012).
- T. Valla, Z.-H. Pan, D. Gardner, Y. S. Lee, and S. Chu, Phys. Rev. Lett. 108, 117601 (2012); T. Schlenk et al., ibid. 110, 126804 (2013); M. Ye et al., Phys. Rev. B 85, 205317 (2012); L. R. Shelford, T. Hesjedal, L. Collins-McIntyre, S. S. Dhesi, F. Maccherozzi, and G. van der Laan, ibid. 86, 081304(R) (2012).
- L. Andrew Wray, Su-Yang Xu, Yuqi Xia, David Hsieh, Alexei V. Fedorov, Yew San Hor, Robert J. Cava, Arun Bansil, Hsin Lin, and M. Zahid Hasan, Nat. Phys. 7, 32 (2011).
- M. R. Scholz, J. Sánchez-Barriga, D. Marchenko, A. Varykhalov, A. Volykhov, L. V. Yashina, and O. Rader, Phys. Rev. Lett. 108, 256810 (2012).
- J. Honolka et al., Phys. Rev. Lett. 108, 256811 (2012).
- M. A. Ruderman and C. Kittel, Phys. Rev. 96, 99 (1954); T. Kasuya, Prog. Theor. Phys. 16, 45 (1956); K. Yosida, Phys. Rev. 106, 893 (1957).
- P. G. de Gennes, J. Phys. Radium 23, 630 (1962).
- A. Y. Zyuzin and B. Spivak, Pis’ma Zh. Eskp. Teor. Fiz. (JETP Lett.) 43, 185 (234) (1986); L. N. Bulaevskii and S. V. Panyukov, ibid., 190 (240) (1986).
- P. A. Lee and A. D. Stone, Phys. Rev. Lett. 55, 1622 (1985).
- A. Jagannathan, E. Abrahams, and M. J. Stephen, Phys. Rev. B 37, 436 (1988); A. Jagannathan, Europhys. Lett. 17, 437 (1992); Stefano Chesi and Daniel Loss, Phys. Rev. B 82, 165303 (2010).
- Eq. (2), which is strictly valid for , reproduces the asymptotic result at for the RKKY interaction in the clean limit, . However, this must not be interpreted as an argument to extrapolate the result to any value of . That becomes evident in the calculation of Eq. (5), for which the result should be independent of the grade of disorder and proportional to . This is only true to the leading order in , as the validity of Eq. (2) ensures. This applies also for the calculation in Eq. (6).
- D. Sherrington and S. Kirkpatrick, Phys. Rev. Lett. 35, 1792 (1975); S. Kirkpatrick and D. Sherrington, Phys. Rev. B 17, 4384 (1978).
- S. F. Edwards and P. W. Anderson, J. Phys. F: Metal Phys. 5, 965 (1975).
- G. Parisi, J. Phys. A. 13, 1101 (1980).
- J. R. L. de Almeida and D. J. Thouless, J. Phys. A 11, 983 (1978).
- Andreas W. W. Ludwig, Matthew P. A. Fisher, R. Shankar, and G. Grinstein, Phys. Rev. B 50, 7526 (1994).
- Paolo Sessi, Felix Reis, Thomas Bathon, Konstantin A. Kokh, Oleg E. Tereshchenko, and Matthias Bode, Nature Commun. 5, 5349 (2014).
- D. K. Efimkin and V. Galitski, Phys. Rev. B 89, 115431 (2014).
- A. H. Castro Neto and B. A. Jones, Phys. Rev. B 62, 14975 (2000).