Interacting spin-3/2 fermions in a Luttinger (semi)metal:competing phases and their selection in the global phase diagram

# Interacting spin-3/2 fermions in a Luttinger (semi)metal: competing phases and their selection in the global phase diagram

Andrs L. Szab Max-Planck-Institut für Physik komplexer Systeme, Nöthnitzer Str. 38, 01187 Dresden, Germany    Roderich Moessner Max-Planck-Institut für Physik komplexer Systeme, Nöthnitzer Str. 38, 01187 Dresden, Germany    Bitan Roy Max-Planck-Institut für Physik komplexer Systeme, Nöthnitzer Str. 38, 01187 Dresden, Germany
July 19, 2019
###### Abstract

We compute the effects of electronic interactions on gapless spin-3/2 excitations that in a noninteracting system emerge at a bi-quadratic touching of Kramers degenerate valence and conduction bands in three dimensions, also known as a Luttinger semimetal. This model can describe the low-energy physics of HgTe, gray-Sn, 227 pyrochlore iridates and half-Heuslers. For the sake of concreteness we only consider the short-range components of the Coulomb interaction (extended Hubbardlike model). By combining mean-field analysis with a renormalization group (RG) calculation (controlled by a “small” parameter , where ), we construct multiple cuts of the global phase diagram of interacting spin-3/2 fermions at zero and finite temperature and chemical doping. Such phase diagrams display a rich confluence of competing orders, among which rotational symmetry breaking nematic insulators and time reversal symmetry breaking magnetic orders (supporting Weyl quasiparticles) are the prominent candidates for excitonic phases. We also show that even repulsive interactions can be conducive to both mundane -wave and topological -wave pairings. The reconstructed band structure (within the mean-field approximation) inside the ordered phases allows us to organize them according to the energy (entropy) gain in the following (reverse) order: -wave pairing, nematic phases, magnetic orders and -wave pairings, at zero chemical doping. However, the paired states are energetically superior over the excitonic ones for finite doping. The phase diagrams obtained from the RG analysis show that for sufficiently strong interactions, an ordered phase with higher energy (entropy) gain is realized at low (high) temperature. In addition, we establish a “selection rule” between the interaction channels and the resulting ordered phases, suggesting that repulsive short-range interactions in the magnetic (nematic) channels are conducive to the nucleation of -wave (-wave) pairing among spin-3/2 fermions. We believe that the proposed methodology can shed light on the global phase diagram of various two and three dimensional interacting multi-band systems, such as Dirac materials, doped topological insulators and the like.

## I Introduction

The discovery of new phases of matter, and the study of the transitions between them, form the core of condensed matter and materials physics chaikin (). Much experimental ingenuity is devoted to realising and controlling different tuning parameters in the laboratory–such as temperature, pressure, magnetic field, chemical composition–to drive a system from one phase to the other chaikin (). Most simply, the appearance of different phases can often be appreciated from the competition between energy and entropy. For example, with decreasing temperature transitions from water vapor to liquid to ice are intimately tied with the reduction of entropy or gain in energy. Similarly, in a metal, entropy of gapless excitations on the Fermi surface is exchanged for condensation energy as a superconducting gap opens tinkham ().

With increasing complexity of quantum materials attained in recent decades, the richness of the global phase diagram of strongly correlated materials has amplified enormously. And various prototypical representatives, such as cuprates, pnictides, heavy fermion compounds display concurrence of competing orders, among which spin- and charge-density-wave, superconductivity, nematicity are the most prominent ones. Besides establishing the existence of – and, hopefully, eventually utilizing in technological applications – these phenomena, an obvious challenge is to discover any simplifying perspective, or at least heuristic classification scheme, for predicting and classifying them: do there exist any organising principles among multiple competing orders that can shed light on the global phase diagram of strongly correlated materials? Restricting ourselves to a specific but remarkably rich metallic system, we here give a partially affirmative answer to this question. We study a collection of strongly interacting spin-3/2 fermions in three dimensions that in the normal phase display a bi-quadratic touching of Kramers degenerate valence and conduction bands at an isolated point in the Brillouin zone, see Fig. 1. This system is also known as Luttinger (semi)metal luttinger (); murakami-zhang-nagaosa ().

Such peculiar quasiparticle excitations can be found for example in HgTe hgte (), gray-Sn gray-sn-1 (); gray-sn-2 (), 227 pyrochlore iridates (LnIrO, where Ln is the lanthanide element) Savrasov (); Balents1 (); Exp:Nakatsuji-1 (); Exp:Nakatsuji-2 (); Exp:armitage () and half-Heusler compounds (such as LnPtBi, LnPdBi) Exp:cava (); Exp:felser (); binghai (). Bi-quadratic band touching has recently been observed in the normal state of PrIrO Exp:Nakatsuji-1 () and NdIrO Exp:Nakatsuji-2 () via angle resolved photo emission spectroscopy (ARPES). While most of the iridium-based oxides support all-in all-out arrangement of 4d Ir-electrons takagi (); krempa-1 (); krempa-2 (); nagaosa (); Balents3 (); yamaji (); troyer (); tokura-1 (); tokura-2 (), a singular member of this family, namely PrIrO, possibly resides at the brink of a metallic spin-ice or 3-in 1-out ordering and supports a large anomalous Hall conductivity AHE-1 (); AHE-2 (); AHE-3 (); goswami-roy-dassarma (). While these materials harbor various competing magnetic orders (due to comparable Hubbard and spin-orbit interactions), smallness of the Fermi surface prohibits the onset of superconductivity at the lowest achievable temperature. On the other hand, half-Heusler compounds accommodate both anti-ferromagentism and unconventional superconductivity Exp:Takabatake (); Exp:Paglione-1a (); Exp:Bay2012 (); Exp:visser-1 (); Exp:Taillefer (); Exp:Zhang (); Exp:visser-2 (); Exp:Paglione-1 (); Exp:Paglione-2 (); Exp:TbPdBi (). A transition between them can be triggered by tuning the de Gennes factor Exp:Paglione-1 (). Moreover, the superconducting YPtBi (with transition temperature K) supports gapless BdG quasiparticles at low temperatures Exp:Paglione-2 (). Therefore, besides its genuine fundamental importance, our quest focuses on a timely issue due to growing material relevance of interacting spin-3/2 fermions, which has triggered a recent surge of theoretical works krempa-1 (); krempa-2 (); nagaosa (); Balents3 (); yamaji (); troyer (); goswami-roy-dassarma (); balents-kim (); Herbut-1 (); lai-roy-goswami (); herbut-1a (); Herbut-2 (); kharitonov (); arago (); BJ-Yang (); Fang2015 (); Yang16 (); brydon-1 (); brydon-2 (); Herbut-3 (); savary-1 (); roy-nevidomskyy (); savary-2 (); sato (); CXLiu-1 (); CXLiu-2 (); brydon-3 (); sunbinLee ().

The effects of electronic interactions on spin-3/2 fermions are addressed within the framework of an extended Hubbardlike model, composed of only the short-range components of repulsive Coulomb interactions. 111In this work we neglect the long-range tail of the Coulomb interaction. When the chemical potential is pinned at the band touching point, long-range Coulomb interaction can give rise to an infrared stable non-Fermi liquid fixed point Abrikosov1 (); Abrikosov2 (); balents-kim (), which, may however be unstable toward the formation of an excitonic phase kohn (); Herbut-1 (); herbut-1a (). By contrast, at finite chemical doping long-range interaction suffers conventional Thomas-Fermi screening and becomes short-ranged in nature. A systematic incorporation of the long-range interaction into our discussion is left for a future investigation. Due to the vanishing density of states (DoS) in a Luttinger system (namely, ), sufficiently weak short-range interactions are irrelevant perturbations. Therefore, any ordering takes place at finite coupling. We here employ a renormalization group (RG) analysis to construct various cuts of the global phase diagram at zero as well as finite chemical doping [see Figs. 3,  4 and  5], and combine it with mean-field analysis to gain insight into the organizing principle among distinct broken-symmetry phases (BSPs) 222All cuts of the global phase diagram are obtained from a RG analysis, accounting for interaction effects on Luttinger fermions. Such an RG analysis can only predict the phase boundaries between the Luttinger (semi)metal and various BSPs. However, our RG analysis is not applicable inside a BSP and cannot capture order-order transitions. The color fillings in Figs. 3,  4,  5,  6,  7 and  16 are implemented solely for the clarity of presentation. . A gist of our findings can be summarized as follows.

1. By computing the reconstructed band structure (within the mean-field approximation) we organize dominant BSPs according to the condensation energy and entropy gains. Results are summarized in Fig. 2. We note that while the stiffness (uniform or anisotropic) of the band gap measures the condensation energy gain, the amount of gapless quasiparticles (resulting in power-law scaling of DoS at low energies) measures the entropy inside the ordered phase. All cuts of the global phase diagram (obtained from a RG analysis) show that the low (high) temperature phase yields larger condensation energy (entropy) gain, see Fig. 3.

2. The quasiparticle spectra inside the -wave paired state and two nematic phases (belonging to the and representations of the cubic or point group) are fully gapped. Hence, nucleation of these three phases leads to the maximal gain of condensation energy, and they appear as the dominant BSPs at zero temperature, as shown in Fig. 3.

3. At finite temperature condensation energy gain competes with the entropy, and phases with higher entropy are realized at higher temperatures. Onset of any nematicity results in an anisotropic gap, in contrast to the situation with an -wave pairing. Thus former orderings are endowed with larger (smaller) entropy (condensation energy), and are found at higher temperatures in Figs. 3, 3. By contrast, the dominant magnetic orders (belonging to the and representations) produce gapless Weyl quasiparticles and result in scaling of the DoS at low energies. Hence, these two magnetic orders carry larger entropy than the nematic phases or the -wave pairing, and can only be found at finite temperature, see Figs. 3, 3. Luttinger semimetal (LSM)- magnetic order (results from the all-in all-out state in pyrochlore lattice Savrasov (); Balents3 ()) transition at finite temperature is consistent with the experimental observation in NdIrO takagi (), while the magnetic ordering (yielding a 3-in 1-out ordering in pyrochlore lattice and supporting anomalous Hall effect goswami-roy-dassarma ()) can be germane for PrIrO AHE-1 (); AHE-2 (); AHE-3 () 333ARPES measurements in NdIrO Exp:Nakatsuji-2 () and PrIrO Exp:Nakatsuji-1 () suggest that the LSM in these compound is isotropic..

4. Local four fermion interactions in the nematic channels are conducive for -wave pairing (at zero and finite chemical doping), see Fig. 3,  3 and 4 , whereas short-range magnetic interactions give birth to topological -wave pairing (only at finite chemical doping), see Fig. 5. We further elaborate such an emergent “selection rule” in Sec. II.2. Confluence of magnetic order and -wave pairing (resulting in gapless BdG quasiparticles, found in YPtBi Exp:Paglione-2 ()) is in (qualitative) agreement with the global phase diagram of LnPdBi Exp:Paglione-1 ().

The theoretical approach outlined in this work is quite general and can be extended to address the effects of electronic interactions in various strongly correlated multi-band systems, among which two-dimensional Dirac and quadratic fermions (respectively relevant for monolayer and bilayer graphene) graphene-RMP (); Balatsky (), three-dimensional doped topological, crystalline and Kondo insulators TI-RMP-1 (); TI-RMP-2 (); TKI-Review (), Weyl materials Weyl-RMP (), twisted bilayer graphene tblg-1 (); tblg-2 (); tblg-3 (), are the most prominent and experimentally pertinent ones. In the future we will systematically study these systems.

### i.1 Outline

The rest of the paper is organized as follows. In the next section we present an extended summary of our main results. The low-energy description of the Luttinger model and its symmetry properties are discussed in Sec. III. In Sec. IV we discuss the reconstructed band structure inside various excitonic and superconducting phases. In Sec. V we introduce the interacting model for spin-3/2 fermions and analyze the propensity toward the formation of various orderings within a mean-field approximation. Sec. VI is devoted to a renormalization group (RG) analysis of interacting Luttinger fermions at zero and finite temperatures and chemical doping. We summarize the main results and highlight some future outlooks in Sec. VII. Additional technical details are relegated to appendices.

## Ii Extended Summary

Our starting point is a collection of spin-3/2 fermions for which the normal state is described by a bi-quadractic touching of Kramers degenerate valence and conduction bands. The corresponding Hamiltonian operator is  luttinger (); murakami-zhang-nagaosa ()

 ^hL(k)=−k22m[cosα3∑j=1^djΓj+sinα5∑j=4^djΓj]−μ, (1)

where s are five -wave harmonics in three dimensions, s are five mutually anticommuting four dimensional Hermitian matrices, and is the effective mass for gapless excitations in the orbitals in a cubic environment. Momentum and chemical potential are measured from the band touching point. Additional details of this model are discussed in Sec. III and Appendix C. The mass anisotropy parameter () lies within the range  goswami-roy-dassarma (). But, for the sake of concreteness we restrict our focus on the isotropic system with . For discussion on the evolution of phase diagrams with varying , see Secs. V.1 and  VI, and Figs. 6 and  16. The LSM is realized as an unstable fixed point at , the red dot in Fig. 1. This fixed point is characterized by the dynamic scaling exponent , determining the relative scaling between energy and momentum according to . The chemical potential is a relevant perturbation at this fixed point, with the scaling dimension . Hence, the correlation length exponent at this fixed point is . Therefore, the LSM can be envisioned as a quantum critical point (QCP) separating electron-doped (for , the brown region) and hole-doped (for , the green region) Fermi liquid phases, as shown in Fig. 1. Our discussion is focused on the quantum critical regime (the shaded region).

The crossover temperature () separating the quantum critical regime accommodating gapless spin-3/2 excitations from the Fermi liquid phases can be estimated in the following way

 T∗∼ℏ22m×1ξ2∼ℏ22m|n|2/3, (2)

where is a characteristic length scale and is the carrier density. Two critical exponents ( and ) and the dimensionality of the system () determine the scaling of various thermodynamic (such as specific heat, compressibility) and transport (such as dynamic conductivity) quantities in this regime.

### ii.1 Critical scaling in noninteracting system

The free-energy density (up to an unimportant temperature () independent constant) inside the critical regime is given by (setting )

 f=T5/2(2m)324π32[Li52(−eμ/T)+Li52(−e−μ/T)], (3)

where represents the polylogarithimic function. The specific heat of this system is given by

 CV=−T∂2f∂T2≈T3/2(2m)3232π32[15a−bμ2T2], (4)

for (ensuring that the system resides inside the critical regime), where and . From the above expression of the free-energy density we can also extract the scaling of compressibility (), given by

 κ=−∂2f∂μ2≈√T(2m)322π32[b+6cμ2T2], (5)

where . Therefore, the presence of finite chemical doping does not alter the leading power-law scaling of physical observables, such as , , but only provides subleading corrections, which are suppressed by a parametrically small quantity 444The scaling of specific heat and compressibility is determined by the dimensionality () and dynamic scaling exponent () according to and , respectively. Also note that

 CV/Tκ≈5.37611 (6)

is a universal ratio, capturing the signature of a quantum critical point in . A detailed derivation of this analysis is shown in Appendix A. Qualitatively similar sub-leading corrections are also found in the scaling of the dynamic conductivity, which we discuss now.

Gauge invariance mandates that the conductivity () must scale as . Hence for a collection of quasiparticles (such as the Luttinger fermions), or in three spatial dimensions. Indeed we find that the Drude (Dr) component of the dynamic conductivity in the Luttinger system is given by [see Appendix B]

 σDr(ω,T)=e2δ(ωT)√mTFDr(μT), (7)

where is a monotonically increasing universal function of its argument [see Eq. (70) and Fig. 21] and is the frequency. On the other hand, the inter-band (IB) component of the optical conductivity reads as

 σIB(ω,T)=e2√mω∑τ=±tanh(ω+2τμ4T). (8)

Hence, inter-band component of the optical conductivity vanishes as as and the LSM can be identified as a power-law insulator.

Therefore, even when the chemical doping is finite there exists a wide quantum critical regime, shown in Fig. 1, where the scaling of thermodynamic and transport quantities are essentially governed by quasiparticles, and the chemical potential provides only sub-leading corrections. Next we highlight the imprint of finite temperature and chemical doping on the global phase diagram of interacting spin-3/2 fermions.

### ii.2 Electron-electron interactions in a Luttinger (semi)metal

In this work we compute the effects of electron-electron interactions on Luttinger fermions, occupying the critical regime of the noninteracting fixed point, see Fig. 1. In this regime any short-range or local four-fermion interaction () is an irrelevant perturbation, since

 [λ]=z−d=2−3=−1,

due to the vanishing DoS, namely . We use a RG analysis, tailored to address the effects of electronic interactions on Luttinger fermions in , constituting a band structure, to arrive at various cuts of the global phase diagram of this system. If, on the other hand, temperature and chemical doping are such that the system resides inside a Fermi liquid phase, the notion of nodal quasiparticles becomes moot and our RG analysis loses its jurisdiction. 555 Note that in the presence of a Fermi surface the DoS is constant and the interaction coupling is dimensionless. Consequently a Fermi liquid becomes unstable toward the formation of a superconductor (often non--wave) even in the presence of repulsive electron-electron interactions, following the spirit of the Kohn-Luttinger mechanism Sankar-RG-RMP (); kohn-Luttinger (); KL-Chubukov-Review (); zanchi-schulz (); hlubina (); raghu-kivelson () and the superconducting transition temperature () mimics the BCS-scaling law . Furthermore, we augment the RG analysis with an organizing principle based on the competition between energy and entropy. To this end we rely on the computation of the reconstructed band structure inside the ordered phases within the mean-field approximation. Subsequently, we also promote a “selection rule” among neighboring phases in the global phase diagram, originating purely from their algebraic or symmetry properties. For the sake of simplicity we concentrate on the isotropic system () in the following three subsections. Nonetheless, our results hold (at least qualitatively) for any arbitrary value of , as summarized in the last subsection.

IIB1. Organizing principle: Emergent Topology & Energy-Entropy

Let us first promote an organizing principle among BSPs according to their contribution to the energy and entropy gain and anticipate their presence in the global phase diagram. In what follows we highlight the reconstructed band structure inside the dominant ordered phases within a mean-field or Hartree-Fock approximation, which by construction undermines the ordered parameter fluctuations. The emergent band topology is computed by diagonalizing an effective single-particle Hamiltonian, composed of the noninteracting Luttinger Hamiltonian and corresponding order parameter, see Sec. IV.3 for details.

Perhaps it is natural to anticipate that at zero temperature strong electronic interactions favor the phases that produce the largest spectral gap, as the onset of these ordered states offers maximal gain of condensation energy. In a LSM there are three candidate BSPs that yield fully gapped quasiparticle spectra: (a) an -wave superconductor, producing a uniform and isotropic gap, and (b) two nematic orders (belonging to the and representations), producing anisotropic gaps. As shown in Fig. 3, only these three phases can be found in an isotropic and interacting LSM at zero temperature.

The energy-entropy competition, leading to an organizing principle among competing phases at finite temperature, can be appreciated from the scaling of the DoS or the stiffness (uniform or anisotropic) of the spectral gap. As mentioned above, the -wave pairing and nematic orders respectively produce uniform and anisotropic gaps, whereas two magnetic orders, belonging to the and representations, respectively produce eight Balents3 () and two goswami-roy-dassarma () isolated simple Weyl nodes, around which the DoS vanishes as for sufficienlty low energies. On the other hand, each copy of the -wave pairings accommodates two nodal loops for which the low-energy DoS scales as [see Sec. IV.4brydon-1 (); Herbut-3 (); roy-nevidomskyy (); savary-2 (); sunbinLee (). Since we are interested in energy or temperature scales much smaller than the ultraviolet cutoff or bandwidth (), the structure of the spectral gap (isotropic or anisotropic) and power-law scaling of DoS carry sufficient information to organize the ordered phases according to their contribution to condensation energy and entropy, summarized in Fig. 2. In brief, existence of more gapless points (resulting in higher DoS near ) yields larger entropy, while a more uniform gap leads to higher gain in condensation energy. From various cuts of the global phase diagram at finite temperature, see Fig. 3, we note the following common feature: Among competing orders, the one with maximal gain in condensation energy appears at low temperature, while the phase with higher entropy is realized at higher temperature, in accordance with the general principle of energy-entropy competition. Since the DoS in a LSM scales as (maximal entropy), it can always be found at sufficiently high (weak) enough temperature (interactions).

A similar conclusion can also be arrived at when the chemical potential is placed away from the band touching point. At finite chemical doping all particle-hole (two nematic and two magnetic) orders produce a Fermi surface (according to the Luttinger theorem Luttinger_Theorem ()) and hence a finite DoS. By contrast, any superconducting order at finite doping maximally gaps the Fermi surface. Therefore, at finite chemical doping superconducting orders are energetically superior to the excitonic orders, and they can be realized at sufficiently low temperature even in the presence of repulsive electronic interactions. Concomitantly, the particle-hole orders are pushed to the higher temperature and interaction regime, see Figs. 4 and  5.

Even though we gain valuable insights into the organization of various BSPs in the global phase diagram of strongly interacting spin-3/2 fermions from the competition between energy and entropy inside the ordered phases (guided by emergent topology of reconstructed band structure), the phase diagrams shown in Figs. 34 and  5 are obtained from an unbiased RG analysis, which systematically accounts for quantum fluctuations beyond the saddle point or mean-field approximation. Next we highlight the key aspects of the RG analysis.

IIB2. Methodology: Renormalization Group

The RG analysis we pursue in this work is controlled by a “small” parameter , measuring the deviation from the lower critical two spatial dimensions () of the theory, where local four-fermion interactions are marginal, with , and hence . Both temperature and chemical potential (bearing the scaling dimension of energy) are relevant perturbations at the fixed point, with scaling dimension . The leading order RG analysis can be summarized in terms of the following set of coupled flow equations

 dtdℓ=zt,dμdℓ=zμ, dgidℓ=−ϵgi+∑j,kgjgkHjk(α,t,μ), (9)

where is the logarithm of the RG scale, and are respectively the dimensionless temperature and chemical potential. For brevity we take in the above flow equations. Here represents the set of dimensionless four-fermion interactions with and are functions of the mass anisotropy parameter , and . The RG flow equations for s are obtained by systematically accounting for quantum corrections to the quadratic order in the s. The relevant Feynman diagrams are shown in Fig. 14. A more detailed discussion of the RG analysis is presented in Sec. VI and Appendix E. Some salient features of the RG analysis are the followings.

1. Temperature () and chemical potential () provide two infrared cutoffs chakravarty (); vafek (), respectively given by

 ℓt∗=1zln(1t(0)),ℓμ∗=1zln(1μ(0)) (10)

for the flow of quartic couplings , where and represent the bare values (). Ultimately stops the flow of four-fermion interactions. At zero temperature and chemical doping the system is devoid of any such natural infrared cutoff, implying .

2. Any weak local four-fermion interaction is an irrelevant perturbation and all orderings (realized when ) take place at finite coupling through quantum phase transitions (QPTs). Such QPTs are controlled by quantum critical points (QCPs) and all transitions are continuous in nature. The universality class of the transition is determined by two critical exponents, given by

 ν−1=ϵ+O(ϵ2)andz=2+O(ϵ), (11)

and for the physically relevant situation .

Using the RG analysis we arrive at various cuts of the global phase diagram of interacting spin-3/2 fermions at (1) zero chemical doping [see Fig. 3] and (2) for finite- [see Figs. 4 and 5]. The universality class of the QPT leaves its signature on the scaling of the transition temperature (). Note that  sondhi-RMP (); sachdev-book (), where is the reduced distance from the critical point, located at . Hence, for and , obtained from the leading order expansion, after setting , irrespective of the choice of the coupling constant and the resulting BSP (see Fig. 17). We discuss this issue in detail in Sec. VI.3.

IIB3. Competing Orders & Selection Rule

The correspondence between a given interaction coupling and the resulting phases can be appreciated by formulating the whole theory in the basis of an eight component Nambu-doubled spinor , introduced in Sec. III.3. In this basis the Luttinger Hamiltonian . Pauli matrices operate on the Nambu or particle-hole index. Any four-fermion interaction takes the form and an order parameter () couples to a fermion bilinear according to , where and are eight dimensional Hermitian matrices. We argue that when is sufficiently strong, it can support only two types of ordered phases, for which 666If and are multi-component vectors of and elements, respectively, then condition (2) is satisfied when at least matrices anti-commute with matrices, where is the ceiling function.

 either(1)^O≡^Ior(2){^O,^I}=0. (12)

This outcome can be appreciated in the following way.

When an interaction coupling diverges toward under coarse graining (indicating onset of a BSP), it provides positive scaling dimension to an order parameter field only when one of the above two conditions is satisfied. We substantiate this argument by considering the relevant Feynman diagrams [see Fig. 15] in Sec. VI.5. Even though we arrive at this “selection rule” among competing orders from a leading order RG calculation, such a simple argument relies on internal symmetries among competing orders (breaking different symmetries) and is expected to hold at the non-perturbative level. We now support this claim by focusing on a specific example.

Let us choose a particular four-fermion interaction (in the nematic channel)

 g13∑j=1(Ψ†Namη3ΓjΨNam)2.

From the phase diagrams shown in Figs. 3 for zero and finite temperature and 4(left) for finite chemical doping, we find that when this coupling constant is sufficiently strong, it supports two distinct BSPs.

1. A nematic order following the representation, for which . In this case selection rule (1) from Eq. (12) is satisfied, since , and hence .

2. An -wave superconductor following the trivial representation, for which , where is a four-dimensional identity matrix. The onset of -wave pairing follows from selection rule (2) in Eq. (12), since .

Moreover we realize that the nematic order and the -wave pairing together constitute an vector, , of five mutually anti-commuting matrices, reflecting the enlarged internal symmetry between these two orders. Following the same spirit we arrive at the following observations.

1. Four fermion interaction in the nematic channel () supports (a) a nematic order transforming under the representation [satisfying selection rule (1)] and (b) an -wave superconductor [satisfying selection rule (2)], as shown in Figs. 3 and  4(right). One can construct an vector by combining these two order-parameters [see Eq. (43)].

2. Four fermion interaction in the channel () supports (a) a magnetic order transforming under the representation [satisfying selection rule (1)], and (b) nematic order and -wave pairings [satisfying selection rule (2)], as shown in Figs. 3 and  5(top). Notice, we can construct multiple copies of composite SU(2) order parameters, by combining the order with nematic or -wave pairing, see Figs. 18(a) and 19.

3. Four fermion interaction in the magnetic channel () supports (a) a magnetic order transforming under the representation [satisfying selection rule (1)], and (b) nematic and -wave pairings [satisfying selection rule (2)], as shown in Figs. 3 and  5(bottom). Combining the magnetic order with nematic or -wave pairing we can construct multiple copies of composite SU(2) vector, see Figs. 18(b),  18(c) and 20.

A more detailed discussion supporting these scenarios is presented in Sec. VI.5. Therefore, combining the energy-entropy competition (obtained from the reconstructed band topology within a mean-field approximation) and an unbiased (controlled by a small parameter ) RG analysis with the selection rule we gain valuable insights into the nature of broken symmetry phases, competing orders and quantum critical phenomena in the global phase diagram of strongly interacting spin-3/2 fermions.

IIB4. Anisotropic Luttinger (semi)metal

So far, we centered our focus on the isotropic system [realized for in Eq. (1)]. Note that for the system enjoys an enlarged (but artificial) SO(3) rotational symmetry. However, in a cubic environment in general. Nonetheless, all the central results we quoted in the last three subsections hold (at least qualitatively) for any arbitrary value of : . The discussion on the role of the mass anisotropy parameter on the global phase diagram of interacting spin-3/2 fermions is rather technical, which we address in depth in Secs. V.1, VI.1, VI.4. We here only quote some key results, which nicely corroborate with the rest of the discussion from this section.

1. We identify the mass anisotropy parameter as a valuable non-thermal tuning parameter, and for suitable choices of this parameter one can find (a) nematic and magnetic order respectively for strong enough and couplings (when ), see Figs. 6 and  6, (b) nematic and magnetic orders for strong enough and (when ), as shown in Figs. 6 and  7 at zero temperature and chemical doping, respectively. These outcomes are in agreement with selection rule (1).

2. At finite chemical doping (a) an -wave pairing emerges from repulsive electronic interaction in the channel [see Fig. 6] as well as channel [see Fig. 6], (b) -wave pairings belonging to the and representations respectively appear for repulsive interaction in the channel [see Fig. 6] and channel [see Fig. 6]. These outcomes are in accordance with selection rule (2), as we argued previously for an isotropic Luttinger system.

We now proceed to a detailed discussion on each component of our work, starting from the noninteracting Luttinger model.

## Iii Luttinger model

We begin the discussion with the Luttinger model describing a bi-quadratic touching of Kramers degenerate valence and conduction bands at an isolated point (here chosen to be the point, for convenience) in the Brillouin zone. In this section, we first present the low-energy Hamiltonian and discuss its symmetry properties (Sec. III.1). Subsequently, we introduce the corresponding imaginary time () or Euclidean action and the notion of the renormalization group (RG) scaling (Sec. III.2). Finally, we define an eight-component Nambu-doubled basis that allows us to capture all, including both particle-hole or excitonic and particle-particle or superconducting, orders within a unified framework (Sec. III.3).

### iii.1 Hamiltonian and Symmetries

The Hamiltonian operator describing a bi-quadratic touching of Kramers degenerate valence and conduction bands in three dimensions is given by luttinger (); murakami-zhang-nagaosa ()

 ^hL(k)=−k2[3∑j=1^dj(^k)2m1Γj+5∑j=4^dj(^k)2m2Γj]−μΓ0, (13)

where is the four-dimensional identity matrix. Chemical potential and momenta are measured from the band touching point. Here, is a five-dimensional unit vector that transforms under the representation under the orbital rotations. Hence is constructed from the -wave form factors or spherical harmonics , as shown in Appendix C. The corresponding four-component spinor basis is given by

 Ψ⊤k=(ck,+32,ck,+12,ck,−12,ck,−32), (14)

where is the fermionic annihilation operator with momenta and spin projection and . The five mutually anti-commuting matrices are defined as

 Γ1=κ3σ2,Γ2=κ3σ1,Γ3=κ3σ0, Γ4=κ1σ0,Γ5=κ3σ3. (15)

Two sets of two dimensional Pauli matrices and respectively operate on the sign () and magnitude () of the spin projections, where . To close the Clifford algebra of all four-dimensional Hermitian matrices we also define ten commutators according to , with and . All sixteen four-dimensional matrices can be expressed in terms of the products of spin-3/2 matrices (), as also shown in Appendix C.

The energy spectra for Luttinger fermions are given by , where for

 Es(k)=k22m ⎷cos2α3∑j=1^d2j+sin2α5∑j=4^d2j, (16)

reflecting the quadratic band touching for , which is protected by the cubic symmetry. For brevity we drop the explicit dependence of on .

Notice that the independence of on manifests the Kramers degeneracy of the valence and conduction bands, ensured by (1) the time-reversal () and (2) the parity or inversion () symmetries. Specifically, under the reversal of time, and and hence , where is the complex conjugation, yielding (reflecting Kramers degeneracy of bands). Under the inversion and .

The “average” mass and the mass anisotropy parameter are respectively given by goswami-roy-dassarma ()

 m=m1m2m1+m2,α=tan−1(m2m1). (17)

Note that for and belong to the (three component) and (two component) representations of the cubic or octahedral () point group, and and are effective masses in these two orbitals, respectively. The mass anisotropy parameter allows us to smoothly interpolate between (1) the limit when the dispersion of the orbital becomes flat, yielding and (2) when the orbital becomes non-dispersive, leading to . For or , the Luttinger model enjoys an enlarged spherical symmetry. Any captures a quadrupolar distortion in the system (still preserving the cubic symmetry). In what follows, we treat as a non-thermal tuning parameter to explore the territory of interacting Luttinger fermions.

The connection between the spin projections ( and ) and the bands can be appreciated most economically by taking . For such a specific choice of momentum axis, the Luttinger Hamiltonian takes a diagonal form, given by

 (18)

From the above expression we can immediately infer that the pseudospin projections on the valence and conduction bands are respectively and .

### iii.2 Lagrangian and Scaling

The imaginary time () Euclidean action corresponding to the non-interacting Luttinger model is given by

 S0=∫dτddxΨ†(τ,x)^hL(k→−i∇)Ψ(τ,x). (19)

The action remains invariant under the following rescaling of space-(imaginary)time coordinates and the fermionic field

 x→eℓx,τ→ezℓτ,Ψ→e−dℓ/2Ψ, (20)

where is the dynamic scaling exponent, measuring the relative scaling between energy and momentum according to . For Luttinger fermions . The parameter is the logarithm of the RG scale. In what follows in Secs. V and VI, we use the above scaling ansatz while addressing the effects of electronic interactions in this system. Under the above rescaling of parameters, the temperature () and chemical potential () scale as

 T→e−zℓT,μ→e−zℓμ. (21)

Therefore, the scaling dimension of these two quantities is (same as that of energy). Throughout, we use the natural unit, in which .

### iii.3 Nambu doubling

To facilitate the forthcoming discussion we here introduce an eight-component Nambu-doubled spinor basis (suitable to capture both exitonic and superconducting orders within a unified framework) according to

 ΨNam=⎡⎣ΨkΓ1Γ3(Ψ†−k)⊤⎤⎦, (22)

where is a four-component spinor, see Eq. (14). In the lower block of we absorb the unitary part of the time-reversal operator , ensuring that the eight-component Nambu spinor () transforms the same way as the original four component spinor under the pseudospin rotation. In this basis the eight-dimensional Luttinger Hamiltonian takes a simple form

 ^hNamL(k)=η3^hL(k), (23)

and the time-reversal operator becomes . The newly introduced set of Pauli matrices operates on the Nambu or particle-hole indices, with . Therefore, by construction while the excitonic orders assume block-diagonal form, all superconducting orders are block-off-diagonal in the Nambu subspace. Note that commutes with the number operator .

## Iv Broken symmetry phases

Next we discuss possible BSPs in this system. We introduce various possible excitonic and superconducting orders in the Nambu basis () in two subsequent sections. Finally, we discuss the reconstructed band structure and emergent topology inside the ordered phases.

### iv.1 Particle-hole or excitonic orders

The effective single-particle Hamiltonian in the presence of all possible momentum-independent or local or intra-unit cell excitonic orders is given by

 Hexclocal=∫d3r(Ψ†Nam^hexclocalΨNam), (24)

where

 ^hexclocal = DensityΔ0η3Γ0+Nematicη3[3∑j=1Δj1Γj+5∑j=4Δj2Γj] + η0[Δ3Γ45+3∑j=1Δj4Γ45Γj+3∑j=15∑k=4Δjk5Γjk]Magnetic.

The ordered phases can be classified according to their transformation under the cubic () point group symmetry. Regular fermionic density () does not break any symmetry (hence does not correspond to any ordering) and transforms under the trivial representation. A three-component nematic order-parameter, constituted by , transforms under the representation. By contrast, a two-component nematic order transforming under the representation is captured by . Both of them break only the cubic symmetry, but preserve time-reversal and inversion symmetries. The ordered phase represents either a time-reversal invariant insulator or a Dirac semimetal, about which more in a moment [see Sec. IV.3]. Since five matrices transform as components of a rank-2 tensor under SO(3) rotations, the two nematic phases represent quadrupolar orders, see Appendix C.

All ordered phases shown in the second line of Eq. (IV.1) break time-reversal symmetry and represent different magnetic orders. For example, corresponds to an octupolar order (since ), transforming under the singlet representation. In a pyrochlore lattice of 227 iridates such an ordered phase represents the “all-in all-out” arrangement of electronic spin between two adjacent corner-shared tetrahedra Savrasov (); Balents3 (). By contrast, “two-in two-out” or “spin-ice” magnetic orderings on a pyrochlore lattice are represented by a three-component vector (accounting for six possible two-in two-out arrangements in a single tetrahedron). Since such an ordered phase contains a linear superposition of dipolar and octupolar moments, and transforms under the representation goswami-roy-dassarma (). Any other magnetic ordering can be represented by a six component vector with and . No physical realization of such multi-component magnetic ordering in any material is currently known, and we do not delve into the discussion on such ordering for the rest of the paper.

### iv.2 Particle-particle or superconducting orders

The effective single particle Hamiltonian in the presence of all possible momentum-independent or local or intra-unit cell superconducting orders reads brydon-1 (); Herbut-3 (); savary-1 (); roy-nevidomskyy ()

 Hpairlocal=∫d3r(Ψ†Nam^hpairlocalΨNam), (26)

where

 ^hpairlocal = (η1cosϕ+η2sinϕ)[s−waveΔpA1gΓ0 (27) + 3∑j=1Δp,jT2gΓj+5∑j=4Δp,jEgΓjd−wave],

and is the global superconducting phase. Any pairing proportional to preserves (breaks) time-reversal symmetry (recall that the time-reversal operator in the Nambu basis is ). Here, is the amplitude of the -wave pairing, transforming under the representation. The -wave pairing breaks only the global symmetry, but preserves the cubic symmetry. On the other hand, captures the amplitude of three -wave pairings (for ) transforming under the representation, and for represents the amplitude of two -wave pairings belonging to the representation. Notice can be expressed in terms of the product of two spin-3/2 matrices, and all five -wave pairings break the cubic symmetry, while introducing a lattice distortion or electronic nematicity in the system. Hence, they stand as representatives of quadrupolar nematic superconductors.

### iv.3 Reconstructed band structure and emergent topology

Next we consider the reconstructed band structure inside different BSPs which provides valuable information regarding the emergent topology inside ordered phases. The onset of any ordering discussed in the previous sections destabilizes the bi-quadratic touching and gives rise to either gapped or gapless quasiparticles (see below). Furthermore, this exercise will allow us to appreciate the energy-entropy competition among different orderings [see Sec. IV.4], which ultimately plays a decisive role in the organization of various phases in the global phase diagram of interacting Luttinger fermions.

1. nematicity: The three component order-parameter for the nematic phase gives birth to gapless quasiparticles for the following four configurations

These four phase lockings are respectively shown as blue, red, green and black points in Fig. 8. The gapless phase corresponds to a topological Dirac semimetal (since nematicity preserves the Kramers degeneracy of valence and conduction bands), similar to the ones recently found in CdAs cdas:Exp () and NaBi nabi:Exp (). The DoS in a Dirac semimetal vanishes as . The Dirac points are located along the body diagonals (the axes) of a cubic system and respectively placed at

 k=±{(1,1,1)1,(1,1,−1)2,(1,−1,1)3,(1,−1,−1)4}k0,

where , as shown in Fig. 8. For any other phase locking within the sector the system becomes an insulator. The spectral gap in the insulating phase is anisotropic and it is energetically superior over the gapless Dirac semimetal phase.

2. nematicity: The two component nematic order is most conveniently described in terms of the following parametrization

 →Δ2=|Δ2|√2(sinϕEg,cosϕEg), (30)

where is the internal angle in the order-parameter space. Only for

 ϕEg=⎧⎪⎨⎪⎩01,2π/32,4π/33⎫⎪⎬⎪⎭ (31)

the quasi-particle spectra are gapless, as shown in Fig. 9, and the ordered phase represents a topological Dirac semimetal. Specifically, for , the Dirac points are respectively located on , and axes (the axes), see Fig. 9, and the separation of two Dirac points is given by , where . For any other phase locking within the sector, the system becomes an insulator.

3. magnet: In the presence of an octupolar ordering, the two-fold degeneracy of the valence and conduction band gets lifted and a pair of Kramers non-degenerate bands touch each other at the following eight points in the Brillouin zone [see Fig. 10]

 k=(±1,±1,±1)k0, (32)

where . They represent simple Weyl points, which act as source (4 of them) and sink (4 of them) of Abelian Berry curvature of unit strength. However, due to an octupolar arrangement of the Weyl nodes, the net Berry curvature through any high-symmetry plane is precisely zero and this phase does not support any anomalous Hall effect. The DoS at low energies then scales as  Savrasov (); Balents3 (); goswami-roy-dassarma ().

4. magnet: For each component of magnetic order (represented by the matrix operator with ) the ordered phase supports two Weyl nodes along one of the axes and a nodal-loop in the corresponding basal plane. For example, when the left and right chiral Weyl nodes are located at where and a nodal-loop is found in the plane, as shown in Fig. 10. Similarly, for and the Weyl nodes are separated along the and axes, and the nodal-loops are respectively found in the and planes. Due to the presence of two Weyl nodes, each configuration of two-in two-out magnetic order supports a finite anomalous Hall effect in the plane perpendicular to the separation of the Weyl nodes. However, any triplet magnetic order, represented by , gets rid of the nodal loop and supports only two Weyl nodes along one of the body-diagonals ( axes). Hence, triplet magnetic orders are energetically favored over their uniaxial counterparts goswami-roy-dassarma ()777The low energy DoS in the presence of a nodal loop and two point nodes (due to a uniaxial order) is dominated by the former and scales as , while in a triplet state the DoS scales as (due to the point nodes). Hence, formation of the triplet ordering causes power-law suppression of the DoS and increases the condensation energy gain.

5. or -wave pairing: Notice that the matrix operator representing an -wave pairing fully anti-commutes with the Luttinger Hamiltonian (for any value of ) and thus corresponds to a mass for Luttinger fermions. The quasiparticle spectra inside the paired state is fully gapped, but the phase is topologically trivial.

6. pairing: Three -wave pairings, proportional to , and matrices, belong to the representation and respectively possess the symmetry of , and pairings. Each component supports two nodal loops in the ordered phase, as shown in the first three rows of Table 1 brydon-1 (); roy-nevidomskyy (). Two nodal loops for the or pairing are shown in Fig. 11. The two nodal loops for or and or pairings can respectively be obtained by rotating the ones shown for pairing by , with respect to the and axes.

7. pairing: pairings proportional to and matrices respectively possess the symmetry of and pairings and each of them supports two nodal loops, as shown in the last two rows of Table 1 brydon-1 (); roy-nevidomskyy (). Note that two nodal loops for the pairing can be realized by rotating the ones for the pairing by about the axis. However, two nodal loops for the pairing, shown in Fig. 11, cannot be rotated into the ones for pairing. Therefore, despite the fact that the and pairings belong to the same representation, they are not energetically degenerate roy-nevidomskyy (); sigrist-RMP (). Since the radius of the nodal loops for the pairing is the smallest, this paired state is the energetically most favorable among five -wave pairings. 888Even though type, such as , pairing can eliminate nodal loops from the spectra in favor of point nodes around which , strong inter-band coupling causes inflation of such nodes and yields Fermi surface of BdG quasiparticles, leading to a constant DoS at lowest energy, followed by  brydon-2 (). Presently, it is not very clear between (a) individual -wave pairing and (b) type pairing, which one is energetically more advantageous.