Nematic order on the surface of a three-dimensional topological insulator

Nematic order on the surface of a three-dimensional topological insulator

Rex Lundgren Department of Physics, The University of Texas at Austin, Austin, Texas 78712, USA Joint Quantum Institute, NIST/The University of Maryland, College Park, Maryland 20742, USA    Hennadii Yerzhakov Department of Physics, University of Alberta, Edmonton, Alberta T6G 2E1, Canada    Joseph Maciejko Department of Physics, University of Alberta, Edmonton, Alberta T6G 2E1, Canada Theoretical Physics Institute, University of Alberta, Edmonton, Alberta T6G 2E1, Canada Canadian Institute for Advanced Research, Toronto, Ontario M5G 1Z8, Canada
August 3, 2019
Abstract

We study the spontaneous breaking of rotational symmetry in the helical surface state of three-dimensional topological insulators due to strong electron-electron interactions, focusing on time-reversal invariant nematic order. Owing to the strongly spin-orbit coupled nature of the surface state, the nematic order parameter is linear in the electron momentum and necessarily involves the electron spin, in contrast with spin-degenerate nematic Fermi liquids. For a chemical potential at the Dirac point (zero doping), we find a first-order phase transition at zero temperature between isotropic and nematic Dirac semimetals. This extends to a thermal phase transition that changes from first to second order at a finite-temperature tricritical point. At finite doping, we find a transition between isotropic and nematic helical Fermi liquids that is second order even at zero temperature. Focusing on finite doping, we discuss various observable consequences of nematic order, such as anisotropies in transport and the spin susceptibility, the partial breakdown of spin-momentum locking, collective modes and induced spin fluctuations, and non-Fermi liquid behavior at the quantum critical point and in the nematic phase.

I Introduction

Rotationally invariant Fermi liquids can spontaneously develop spatial anisotropy as a result of strong electron-electron interactions, a possibility first considered by Pomeranchuk pom (). In the simplest scenario, for sufficiently strong attractive interactions in the angular momentum channel the ground state energy of the Fermi liquid is lowered by a spontaneous quadrupolar distortion of the Fermi surface, leading to transport anisotropies and non-Fermi liquid behavior Oganesyan et al. (2001). Alternatively, the resulting time-reversal and translationally invariant form of order, nematic order, can arise via thermal or quantum melting of translational symmetry-breaking stripe/smectic orders Fradkin et al. (2010). There is strong experimental evidence for the existence of a nematic phase in quantum Hall states Lilly et al. (1999a); Du et al. (1999); Pan et al. (1999); Lilly et al. (1999b); Cooper et al. (2002); Xia et al. (2010, 2011); Samkharadze et al. (2016), high-temperature superconductors Vojta (2009); Fernandes et al. (2014); Kuo et al. (2016), and SrRuO Borzi et al. (2007); Puetter et al. (2010). On the theory side, nematic order has been studied in a wide variety of systems including quantum Hall states Fradkin and Kivelson (1999); Fradkin et al. (2000); Wexler and Dorsey (2001); Wexler and Ciftja (2002); Radzihovsky and Dorsey (2002); Doan and Manousakis (2007); Mulligan et al. (2010, 2011); Maciejko et al. (2013); You and Fradkin (2013); You et al. (2014, 2016); Regnault et al. (), graphene Lamas et al. (2009); Valenzuela and Vozmediano (2008), two- and three-dimensional systems with quadratic band crossing Sun et al. (2009); Janssen and Herbut (2015), three-dimensional Dirac semimetals Zhang et al. (2016), dipolar Fermi gases Fregoso and Fradkin (2009); Fregoso et al. (2009); Chan et al. (2010), high-temperature superconductors Kivelson et al. (2004) and doped Mott insulators Kivelson et al. (1998).

The surface of 3D topological insulators offers a new type of gapless matter, the 2D helical Dirac fermion, which differs qualitatively from conventional Fermi systems due to the phenomenon of spin-momentum locking Hasan and Kane (2010); Qi and Zhang (2011). This begs the question whether criteria for electronic instabilities and the nature of possible broken-symmetry states on the surface of a 3D topological insulator differ from those of conventional 2D Fermi systems. While previous work has focused largely on superconducting Santos et al. (2010); Roy et al. (2013); Nandkishore et al. (2013); Das Sarma and Li (2013); Grover et al. (2014); Ponte and Lee (2014); Neupert et al. (2015); Witczak-Krempa and Maciejko (2016); Zerf et al. (2016); Jian et al. (2016) and time-reversal breaking Xu (2010a, b); Kim and Takimoto (2011); Jiang and Wu (2011); Ghaemi and Ryu (2012); Nogueira and Eremin (2013); Sitte et al. (2013); Bahri and Potter (2015); Mendler et al. (2015); Neupert et al. (2015); Gorbar et al. (2016) instabilities, little attention has been devoted to nematic instabilities, with the exception of Ref. Roy et al. (2015) which studies the spontaneous breaking of a discrete rotation symmetry on the surface of a topological Kondo insulator with multiple Dirac cones.

Our focus here is the isotropic-nematic phase transition on the surface of a 3D topological insulator with a single rotationally invariant Dirac cone. For an undoped system (chemical potential at the Dirac point) one always has continuous rotational invariance in the low-energy limit; for a doped system our theory could apply to a number of experimentally realized topological insulators with very nearly circular Fermi surfaces, such as BiSe Xia et al. (2009); *hsieh2009; *pan2011, BiTeSe Neupane et al. (2012); *neupane2013, SbBiSeTe Neupane et al. (2012), BiSbTeSe Kim et al. (2014), TlBiSe Kuroda et al. (2015), strained -Sn on InSb(001) Barfuss et al. (2013), and strained HgTe Crauste et al. (). A phenomenological Landau Fermi liquid theory of the topological surface state developed earlier by two of us showed that an isotropic-nematic quantum phase transition can occur in the doped system for a sufficiently negative value of the “projected” Landau parameter  Lundgren and Maciejko (2015), in full analogy with the standard Pomeranchuk instability. In this work we construct a field theory of the transition, investigate both the doped and undoped limits, and extend our analysis to nonzero temperatures. In the doped limit we find a continuous transition already at zero temperature, with a breakdown of helical Fermi liquid behavior at the quantum critical point and in the nematic phase, in analogy with the spin-degenerate problem Oganesyan et al. (2001). The nematic phase exhibits a partial breakdown of spin-momentum locking, in the sense that spin and momentum are no longer orthogonal to each other except at certain discrete points on the Fermi surface. Other unusual observable consequences of the spin-orbit coupled nature of nematic order in this system include anisotropy in the in-plane spin susceptibility in the absence of time-reversal symmetry breaking and the generation of spin fluctuations from nematic fluctuations at finite frequency. At zero doping the isotropic-nematic transition is first-order at zero temperature and becomes continuous at a finite-temperature tricritical point.

The paper is organized as follows. In Sec. II, we introduce our model and argue that strong spin-orbit coupling on the topological insulator surface warrants a novel type of nematic order parameter that mixes charge and spin degrees of freedom. In Sec. III, we construct a mean-field theory of the isotropic-nematic transition at both zero and finite temperature and discuss the consequences of nematic order for electronic properties at the mean-field level. Sec. IV discusses fluctuation effects beyond mean-field theory, namely, collective modes and their effect on electronic properties. A brief conclusion is given in Sec. V.

Ii Model and Nematic Order Parameter

In this section, we introduce our field-theoretic model for the isotropic-nematic transition on the surface of a 3D topological insulator. We follow largely the approach of Ref. Oganesyan et al. (2001), with important caveats due to the presence of strong spin-orbit coupling, as will be seen below. While nematic order in 2D electron gases with Rashba spin-orbit coupling has been studied before Berg et al. (2012); Ruhman and Berg (2014), such systems have two degenerate concentric Fermi surfaces and are thus qualitatively distinct from the single, nondegenerate helical Fermi surface considered here.

The Hamiltonian that describes the noninteracting gapless surface state of a topological insulator with a single Dirac cone is given by Hasan and Kane (2010); Qi and Zhang (2011) (in units where )

(1)

where is a two-component Dirac spinor, is the Fermi velocity, is the chemical potential, and

(2)

where is a vector of Pauli matrices, and . The Hamiltonian (1) has a continuous spatial rotation symmetry, , where

(3)

is the component of total angular momentum.

In order to study the isotropic-nematic transition we need a suitable microscopic definition of the nematic order parameter in terms of the fermionic fields . In general, nematic order is described by a quadrupolar order parameter which transforms as a real, traceless symmetric rank-two tensor under rotations de Gennes and Prost (1995). Because of spin-orbit coupling, here the relevant rotations are simultaneous rotations in real space and spin space, generated by the total angular momentum (3). Therefore, unlike for spin rotationally invariant Fermi liquids Oganesyan et al. (2001) the nematic order parameter can involve both the spatial (charge) and spin degrees of freedom of the electron. To lowest order in the electron momentum, the appropriate generalization of the nematic order parameter considered in Ref. Oganesyan et al. (2001) for spin rotationally invariant Fermi liquids to the surface state of 3D topological insulators is

(4)

where , and is a vector of symmetrized derivatives whose action is defined as . This ensures is a Hermitian operator. Finally, the parameter is defined differently depending on whether one is in the doped or undoped limit. We consider that four-fermion interactions, to be written out explicitly below, only act within a high-energy cutoff that can be converted to a momentum cutoff by dividing by . In the undoped limit , we define and the order parameter is local in space. This order parameter was first introduced by one of us in the context of nematic instabilities of the Majorana surface state of superfluid He- Park et al. (2015), and its 3D analog was proposed as an order parameter for parity-breaking phases of spin-orbit coupled bulk metals Fu (2015); Norman (2015). In the doped limit, defined as , only (angular) degrees of freedom on the Fermi surface are relevant and we define  Wu et al. (2007).

In the spirit of Ref. Oganesyan et al. (2001), we consider an attractive four-fermion interaction in the quadrupolar () channel,

(5)

where denotes a trace over the spatial (nematic) indices . The action in imaginary time is then

(6)

As our focus is the vicinity of the isotropic-nematic transition, interactions in other angular momentum channels have been ignored. Indeed, in the doped limit, as long as such interactions are less than the critical value for a Pomeranchuk instability, they will simply lead to a finite renormalization of physical quantities such as the Fermi velocity Lundgren and Maciejko (2015). While the phenomenological Landau Fermi liquid description does not strictly apply to the undoped case, we will assume in this case that interactions in channels are sufficiently weak so there are no competing instabilities.

Iii Mean-Field Theory

To investigate a possible isotropic-nematic phase transition in the action (6), we analyze it in the mean-field approximation. Introducing a real auxiliary scalar field to decouple the four-fermion term via the Hubbard-Stratonovich transformation, we have

(7)

Assuming a uniform and static order parameter , and integrating out the fermions, we obtain the following saddle-point free energy density,

(8)

where , is a fermionic Matsubara frequency. We have rotated the order parameter such that , without loss of generality (corresponding to the principal axes of the distorted Fermi surface being parallel to the and axes 111There is a angle difference between the naive orientation of in Eq. (4) and the principal axes of the distorted Fermi surface, or equivalently, the orientation of the effective spinless nematic order parameter that results from projection to the Fermi surface Lundgren and Maciejko (2015).), and

(9)

is the mean-field dispersion relation of fermionic quasiparticles in the nematic phase (for ), where is the dispersion relation in the isotropic phase. This corresponds to an anisotropic Dirac cone (in the doped limit, is only meant to model the dispersion of quasiparticles on the Fermi surface, with ). Here is to be understood in momentum space, i.e., in the undoped limit and in the doped limit. Performing the sum over Matsubara frequencies, and ignoring constant terms, we obtain

(10)

where corresponds to the upper and lower branches of the Dirac cone, respectively, and we have taken the infinite volume limit . At zero temperature, Eq. (10) becomes the ground state energy density,

(11)

In the following our analysis is performed at constant .

iii.1 Undoped limit

We first evaluate the free energy density in the undoped limit (). At zero temperature, we have

(12)

where we have imposed the momentum cutoff . The integral over momentum can be performed exactly, and we obtain

(13)

where is the complete elliptic integral of the second kind, and we define a dimensionless nematic order parameter and a dimensionless interaction strength . A strongly first-order isotropic-nematic transition is found at a critical value , with a jump of order one in the order parameter at the transition, corresponding to a value of on the order of the high-energy cutoff . This is to be expected since has units of energy, and in the undoped limit the only energy scale in the problem is the cutoff (the critical value of the interaction strength is also determined by the cutoff, since the interaction (5) is perturbatively irrelevant at the Dirac point). Expanding (13) in powers of in the limit , we find

(14)

hence the limit of metastability of the isotropic phase (corresponding to the divergence of the nematic susceptibility) is , but this is preempted by the first-order transition at . The limit of metastability of the nematic phase can be found numerically, and is .

Figure 1: First-order isotropic-nematic quantum phase transition in the undoped limit (). Plots of the mean-field ground state energy density in units of are given as a function of the dimensionless nematic order parameter , for (blue curve), (black curve), and (red curve), where is the dimensionless interaction strength with critical value at the transition. The leading correction to linear dispersion is given by .
(a)
(b)
Figure 2: (color online) Finite-temperature isotropic-nematic transition in the undoped limit (): (a) Jump in dimensionless nematic order parameter at the first-order phase transition as a function of temperature; (b) Mean-field phase diagram in the plane of of temperature and dimensionless interaction strength . A first-order transition (red line) at low temperature turns into a continuous transition (blue line) above a tricritical point (black dot). Dotted lines correspond to limits of metastability of the isotropic () and nematic () phases.

The magnitude of the order parameter jump at the transition can be reduced somewhat by considering the effects of nonzero band curvature at the Dirac point, i.e., deviations from a perfectly linear dispersion (which are present anyway in real topological insulator materials). In other words, we replace in the noninteracting dispersion by a -dependent Fermi velocity

(15)

with the dimensionless parameter representing the leading correction. Such corrections are formally irrelevant in the low-energy limit but affect the free energy Oganesyan et al. (2001); You et al. (2014), which depends on the noninteracting dispersion at all wavevectors up to the cutoff. In the presence of such terms the energy density cannot be evaluated analytically and one must resort to numerical integration. A typical plot of the ground state energy density in the vicinity of the transition for nonzero is given in Fig. 1. We have found that negative values of reduce both the critical interaction strength and order parameter jump at the transition below their values for a strictly linear dispersion.

The apperance of a first-order transition is somewhat surprising, since Landau theory predicts a continuous isotropic-nematic transition in 2D (unlike in 3D, there are no cubic invariants). Expanding the quasiparticle dispersion relation in powers of in Eq. (12), and performing the integral over , we obtain the Landau theory

(16)

where for all . We have checked that the only way to get a quartic term with positive coefficient is to consider a -dependent Fermi velocity that becomes negative at a certain value of below the cutoff , in clear contradiction with the assumption of a single Dirac point in the low-energy spectrum. Therefore, the Landau theory (16) is unbounded from below for sufficiently large , in disagreement with the exact energy density (13) which behaves qualitatively like in Fig. 1. As a result, there must be nonanalytic terms in Eq. (13), but missed by the Landau expansion around , that stabilize the energy density. Such nonanalytic terms are ultimately responsible for the first-order character of the phase transition. In fact, for the energy density (13) becomes

(17)

Thus the energy density is stabilized at large by the “bare” (tree-level) mass term , which grows faster than the negative term coming from the one-loop fermion determinant, i.e., the integral over quasiparticle energies in Eq. (12). The latter is in fact negative for all . We note that a first-order Ising nematic transition at zero temperature was also found for a model of interacting electrons on the square lattice Khavkine et al. (2004). In this case van Hove singularities in the quasiparticle density of states, corresponding to Lifshitz transitions tuned by the value of , are responsible for nonanalyticities in the energy density and the first-order character of the transition.

At finite temperature the free energy density in the undoped limit is given by

(18)

In the remainder of this section we focus on the limit of strict linear dispersion . The integral over the magnitude of can be evaluated analytically in terms of dilogarithms and trilogarithms ; the remaining angular integral must be performed numerically. In Fig. (a)a we plot the jump in the order parameter at the transition as a function of temperature . The jump decreases smoothly from its value at zero temperature, eventually vanishing above a certain temperature corresponding to a tricritical point; for the transition is continuous (a similar behavior was found in Ref. Khavkine et al. (2004)). Since vanishes at the tricritical point, to find we expand the free energy density (10) in powers of . To describe the tricritical point we must expand to sixth order,

(19)

where are functions of . We find that for , which comprises the tricritical point (Fig. (a)a). The tricritical point is found from the condition , from which we find and . The finite-temperature phase diagram is shown in Fig. (b)b, in which we also plot the limits of metastability of the isotropic () and nematic () phases. Note that the first-order phase boundary and limits of metastability are obtained from the numerically evaluated, exact free energy density (10) rather than from the Landau expansion (19), which is accurate only in the vicinity of the continuous transition. Strictly speaking, the finite-temperature phase transition for is a Kosterlitz-Thouless transition and the nematic phase only exhibits quasi-long-range order at finite (but is truly long-range ordered at ).

At the mean-field level, the nematic phase is a theory of noninteracting Dirac quasiparticles with anisotropic dispersion, with Hamiltonian where

(20)

Without loss of generality we choose , , and thus

(21)

The velocities in the and directions (i.e., parallel to the principal axes of the nematic order parameter) at the Dirac point are

(22)

Away from , the density of states remains linear near the Dirac point, . In the limit of strict linear dispersion , the value () thus corresponds to a Lifshitz transition where the quasiparticle dispersion vanishes along () and degenerates into the intersection of two planes, i.e., a quasi-1D Dirac dispersion with formally infinite density of states. In the presence of nonzero band curvature however [Eq. (15)], this degeneracy is lifted, and the flat direction acquires a cubic dispersion at small momenta,

(23)

with and interchanged for . This corresponds to a density of states of the form near the Dirac point .

An interesting signature of the unusual type of nematic order described here is anisotropy in the in-plane spin susceptibility in the absence of any time-reversal symmetry breaking. To compute the spin susceptibility we augment the mean-field Hamiltonian matrix (33) with a Zeeman term,

(24)

where is the -factor, is the Bohr magneton, and is an in-plane magnetic field. To linear order in , we find

(25)

where is the spin susceptibility tensor at temperature , is the dimensionless nematic order parameter at temperature , and is a smooth function of temperature (Fig. 3) defined as

(26)

Thus anisotropy in the in-plane susceptibility is a direct measure of nematic order. For , the transition is continuous (blue curve in Fig. (b)b) thus is small near and the expression (25) can be used in the vicinity of the transition. We thus expect

(27)

on the nematic side of the transition, for . Thus the susceptibility anisotropy can give a direct measure of the order parameter critical exponent , which is in mean-field theory. In the first-order region, since may not be small Eq. (25) cannot be directly used, but we nonetheless expect the anisotropy to be nonzero everywhere in the nematic phase and to vanish in the isotropic phase.

From a qualitative standpoint, the observation of in-plane spin susceptibility anisotropy in the absence of time-reversal symmetry breaking distinguishes the unusual type of nematic order discussed here from other types of order. For conventional nematic order in spin rotationally invariant systems Oganesyan et al. (2001), the breaking of rotation symmetry is in the charge sector and does not cause anisotropy in the spin sector. In-plane ferromagnetic order would lead to anisotropy in the spin response, but requires time-reversal symmetry breaking.

Figure 3: Plot of the dimensionless function defined in Eq. (26).

iii.2 Doped limit

In the doped limit , the cutoff is imposed around the Fermi surface,

(28)

where is the (isotropic) Fermi momentum of noninteracting electrons. We obtain the ground state energy density (11) to leading order in as

(29)

where is the noninteracting density of states at the Fermi surface. Since the coefficient of the term is positive, we therefore find a continuous quantum phase transition at a critical value of the interaction strength given by

(30)

From general considerations we expect a line of finite- Kosterlitz-Thouless phase transitions that terminates at this quantum critical point. We note also that Eq. (30) corresponds precisely to the Pomeranchuk criterion

(31)

derived from a phenomenological Landau theory for the helical Fermi liquid on the surface of a 3D topological insulator Lundgren and Maciejko (2015). In this context the dimensionless “projected” Landau parameters are defined as for , where is the quasiparticle interaction strength in angular momentum channel . The difference in sign arises simply from the fact that in Eq. (5) an attractive interaction corresponds to , while in Ref. Lundgren and Maciejko (2015) it corresponds to .

A first observable signature of nematic order of the type we have described in the doped limit is the partial breakdown of spin-momentum locking. In the doped limit, the mean-field Hamiltonian for fermionic quasiparticles is where

(32)

and . Without loss of generality we choose , , and thus

(33)

Eq. (33) describes an anisotropic Fermi surface. Near the Fermi surface, the eigenstates have positive helicity (assuming , thus above the Dirac point) and are given by

(34)

where we define

(35)

We introduce a new dimensionless order parameter for the doped limit. The expectation value of the spin operator on the Fermi surface is in plane, with components

(36)

thus nematic order affects the spin polarization on the Fermi surface. To leading order in , the angle between the spin vectors in the presence and absence of nematic order is

(37)

Thus except for four points on the Fermi surface , spin and momentum are no longer orthogonal (Fig. 4). However, one might naively think that spin-momentum locking is preserved in the sense that the spin vector remains tangent to the Fermi surface even if the latter is distorted. This is not true: defining a unit vector tangent to the distorted Fermi surface (that winds around the Fermi surface clockwise), we have

(38)

to leading order in , thus the spin vector is tangent to the Fermi surface only at four points, (Fig. 4). This partial breakdown of spin-momentum locking except at high-symmetry points could be detected experimentally using spin-resolved angle-resolved photoemission spectroscopy (ARPES), using for instance the setups described in Ref. Xia et al. (2009); *hsieh2009; *pan2011.

Figure 4: (color online) Partial breakdown of spin-momentum locking in the nematic phase. Blue dashed line: Fermi surface in the isotropic phase (); orange dashed line: Fermi surface in the nematic phase (here shown for ). The red (black) vectors represent the expectation value of spin on the Fermi surface in the isotropic (nematic) phase. Except at four special momenta (green dots), spin in the nematic phase is no longer perpendicular to momentum, nor is it tangential to the (distorted) Fermi surface.

As in the undoped case, nematic order of the type considered here would lead to anisotropy in the in-plane spin susceptibility. Here the transition is continuous already at zero temperature, and in the vicinity of the zero temperature quantum critical point we find

(39)

to leading order in . More conventional measures of nematicity, such as anisotropy in the in-plane resistivity Oganesyan et al. (2001); Kee et al. (2003), apply here as well. Considering scattering on nonmagnetic impurities modelled by a collision time , a calculation of the conductivity using the Kubo formula and impurity-averaged Green’s functions in the first Born approximation gives

(40)

to leading order in and assuming weak disorder . By symmetry we anticipate an analogous result in the undoped case.

Iv Fluctuation effects

We now go beyond the mean-field level and investigate the effect of fluctuations in the vicinity of the quantum critical point in the doped limit . Following Ref. Oganesyan et al. (2001), we rewrite the order parameter in terms of the Pauli matrices and ,

(41)

where

(42)

and we define in the sense of Fourier transforms (see Eq. (4)). We can now rewrite the imaginary-time action in a vectorial form,

(43)

where and

(44)

is the noninteracting Green’s operator. Introducing a bosonic auxiliary field to decouple the four-fermion term, we have

(45)

After integrating out the fermions to second order in , we find the effective action

(46)

where the inverse propagator for the auxiliary field is given to lowest order in momentum and Matsubara frequency by

(47)

Here is the distance from criticality which gives a mass to the auxiliary field, gives it a finite velocity, and

(48)

is a dynamical term where and is the angle between and the axis. Performing the integral over , we have

(49)

which, after a rotation of by , gives the same inverse propagator as for the spinless nematic Fermi fluid Oganesyan et al. (2001). The effective action (46) can be diagonalized by a rotation , , where

(50)

Here is an orthogonal rotation matrix and . Thus and correspond to the longitudinal and transverse components of , respectively. The transformed inverse propagator is

(51)

For small , we have

(52)
(53)

iv.1 Collective modes

Since the inverse propagator of nematic fluctuations is the same as in the spinless case, the number and dispersion of collective modes, given by the condition

(54)

is also the same. Analytically continuing Eq. (52)-(53) to real frequencies , we find

(55)
(56)

to leading order in . At criticality , the collective mode dispersions are

(57)

thus is an undamped mode and is an overdamped mode. Since in the long-wavelength limit , the overdamped mode dominates the long-wavelength response and the dynamical critical exponent at the transition is  Oganesyan et al. (2001).

We note that although corresponds to longitudinal fluctuations of , when projecting to the Fermi surface the longitudinal (11 and 22) components of the order parameter (4) map to the transverse (12 and 21) components of the usual spinless nematic order parameter

(58)

where the effectively spinless field creates electrons of the appropriate helicity on the Fermi surface (see Sec. S4 of the Supplemental Material in Ref. Lundgren and Maciejko (2015)). Likewise, under projection the transverse components of (4) are mapped to the longitudinal components of (58). Thus in this sense () is the transverse (longitudinal) mode, in accordance with the terminology of Ref. Oganesyan et al. (2001).

In the nematic phase (), we consider Gaussian fluctuations about the classical saddle point, which we take to be without loss of generality. Near the critical point where is small, the leading change in the effective action for fluctuations compared to the isotropic phase is to the uniform and static part () of the inverse propagator Oganesyan et al. (2001),

(59)

i.e., the longitudinal (amplitude) mode acquires a mass and the transverse (Goldstone) mode is massless. Deep in the nematic phase (i.e., not small), the part and the dynamical part will be modified from their form at , but our conclusions drawn from the small limit will not be affected in a major way (for instance, a finite would lead to a difference in stiffness for the amplitude and Goldstone modes). The two eigenvalues and of the inverse propagator (59) give the spectrum of collective modes in the nematic phase. The inverse transverse propagator, given by

(60)

corresponds to the gapless nematic Goldstone mode, which is overdamped due to Landau damping except along the principal axes of the distorted Fermi surface ( for the saddle point considered, corresponding to ). Along those directions the inverse transverse propagator reduces to Eq. (52) and the Goldstone mode disperses quadratically according to in Eq. (57). Those undamped directions also correspond to the Fermi surface momenta where spin-momentum locking is preserved (green dots in Fig. 4). The inverse longitudinal propagator is given by

(61)

and describes gapped amplitude fluctuations, as expected.

Despite the number and dispersion of collective modes being formally the same as in the spinless nematic Fermi fluid, their physical nature is very different: in the latter case only charge degrees of freedom fluctuate, while fluctuations of the spin-orbit-coupled nematic order parameter (4) strongly mix charge and spin. An important observable consequence of this difference is that nematic fluctuations in the helical liquid considered here should strongly couple to the spin sector. While static nematic order does not break time-reversal symmetry and thus cannot induce a static spin polarization, nematic fluctuations can in principle induce spin fluctuations. To quantify this effect, one can use linear response: a nematic fluctuation with momentum and frequency should induce a spin fluctuation with the same momentum and frequency,

(62)

if a suitably defined retarded spin-nematic susceptibility is nonzero. An appropriate definition is

(63)

in real space and time, where is the spin operator and is the operator that couples to nematic fluctuations in Eq. (45). Eq. (63) will differ in the isotropic and nematic phases; here we compute in the isotropic phase and find a nonzero result, but we expect a nonzero result in the nematic phase as well.

Figure 5: One-loop diagrams for (a) the spin-nematic susceptibility [Eq. (IV.1)]; (b) the electron self-energy [Eq. (70)].

In the Matsubara frequency domain, the spin-nematic susceptibility is given by the bubble diagram in Fig. 5(a),

(64)

where

(65)
(66)

are the Fourier transform of the nematic vertices (42) and is the unperturbed electron Green’s function, given by

(67)

The retarded spin-nematic susceptibility is obtained from (IV.1) by analytic continuation . We evaluate its imaginary part at zero temperature and in the long-wavelength , low-energy limits. To leading order in , we find