Interacting spin3/2 fermions in a Luttinger (semi)metal:
competing phases and their selection in the global phase diagram
Abstract
We compute the effects of electronic interactions on gapless spin3/2 excitations that in a noninteracting system emerge at a biquadratic touching of Kramers degenerate valence and conduction bands in three dimensions, also known as a Luttinger semimetal. This model can describe the lowenergy physics of HgTe, graySn, 227 pyrochlore iridates and halfHeuslers. For the sake of concreteness we only consider the shortrange components of the Coulomb interaction (extended Hubbardlike model). By combining meanfield analysis with a renormalization group (RG) calculation (controlled by a “small” parameter , where ), we construct multiple cuts of the global phase diagram of interacting spin3/2 fermions at zero and finite temperature and chemical doping. Such phase diagrams display a rich confluence of competing orders, among which rotational symmetry breaking nematic insulators and time reversal symmetry breaking magnetic orders (supporting Weyl quasiparticles) are the prominent candidates for excitonic phases. We also show that even repulsive interactions can be conducive to both mundane wave and topological wave pairings. The reconstructed band structure (within the meanfield approximation) inside the ordered phases allows us to organize them according to the energy (entropy) gain in the following (reverse) order: wave pairing, nematic phases, magnetic orders and wave pairings, at zero chemical doping. However, the paired states are energetically superior over the excitonic ones for finite doping. The phase diagrams obtained from the RG analysis show that for sufficiently strong interactions, an ordered phase with higher energy (entropy) gain is realized at low (high) temperature. In addition, we establish a “selection rule” between the interaction channels and the resulting ordered phases, suggesting that repulsive shortrange interactions in the magnetic (nematic) channels are conducive to the nucleation of wave (wave) pairing among spin3/2 fermions. We believe that the proposed methodology can shed light on the global phase diagram of various two and three dimensional interacting multiband systems, such as Dirac materials, doped topological insulators and the like.
I Introduction
The discovery of new phases of matter, and the study of the transitions between them, form the core of condensed matter and materials physics chaikin (). Much experimental ingenuity is devoted to realising and controlling different tuning parameters in the laboratory–such as temperature, pressure, magnetic field, chemical composition–to drive a system from one phase to the other chaikin (). Most simply, the appearance of different phases can often be appreciated from the competition between energy and entropy. For example, with decreasing temperature transitions from water vapor to liquid to ice are intimately tied with the reduction of entropy or gain in energy. Similarly, in a metal, entropy of gapless excitations on the Fermi surface is exchanged for condensation energy as a superconducting gap opens tinkham ().
With increasing complexity of quantum materials attained in recent decades, the richness of the global phase diagram of strongly correlated materials has amplified enormously. And various prototypical representatives, such as cuprates, pnictides, heavy fermion compounds display concurrence of competing orders, among which spin and chargedensitywave, superconductivity, nematicity are the most prominent ones. Besides establishing the existence of – and, hopefully, eventually utilizing in technological applications – these phenomena, an obvious challenge is to discover any simplifying perspective, or at least heuristic classification scheme, for predicting and classifying them: do there exist any organising principles among multiple competing orders that can shed light on the global phase diagram of strongly correlated materials? Restricting ourselves to a specific but remarkably rich metallic system, we here give a partially affirmative answer to this question. We study a collection of strongly interacting spin3/2 fermions in three dimensions that in the normal phase display a biquadratic touching of Kramers degenerate valence and conduction bands at an isolated point in the Brillouin zone, see Fig. 1. This system is also known as Luttinger (semi)metal luttinger (); murakamizhangnagaosa ().
Such peculiar quasiparticle excitations can be found for example in HgTe hgte (), graySn graysn1 (); graysn2 (), 227 pyrochlore iridates (LnIrO, where Ln is the lanthanide element) Savrasov (); Balents1 (); Exp:Nakatsuji1 (); Exp:Nakatsuji2 (); Exp:armitage () and halfHeusler compounds (such as LnPtBi, LnPdBi) Exp:cava (); Exp:felser (); binghai (). Biquadratic band touching has recently been observed in the normal state of PrIrO Exp:Nakatsuji1 () and NdIrO Exp:Nakatsuji2 () via angle resolved photo emission spectroscopy (ARPES). While most of the iridiumbased oxides support allin allout arrangement of 4d Irelectrons takagi (); krempa1 (); krempa2 (); nagaosa (); Balents3 (); yamaji (); troyer (); tokura1 (); tokura2 (), a singular member of this family, namely PrIrO, possibly resides at the brink of a metallic spinice or 3in 1out ordering and supports a large anomalous Hall conductivity AHE1 (); AHE2 (); AHE3 (); goswamiroydassarma (). While these materials harbor various competing magnetic orders (due to comparable Hubbard and spinorbit interactions), smallness of the Fermi surface prohibits the onset of superconductivity at the lowest achievable temperature. On the other hand, halfHeusler compounds accommodate both antiferromagentism and unconventional superconductivity Exp:Takabatake (); Exp:Paglione1a (); Exp:Bay2012 (); Exp:visser1 (); Exp:Taillefer (); Exp:Zhang (); Exp:visser2 (); Exp:Paglione1 (); Exp:Paglione2 (); Exp:TbPdBi (). A transition between them can be triggered by tuning the de Gennes factor Exp:Paglione1 (). Moreover, the superconducting YPtBi (with transition temperature K) supports gapless BdG quasiparticles at low temperatures Exp:Paglione2 (). Therefore, besides its genuine fundamental importance, our quest focuses on a timely issue due to growing material relevance of interacting spin3/2 fermions, which has triggered a recent surge of theoretical works krempa1 (); krempa2 (); nagaosa (); Balents3 (); yamaji (); troyer (); goswamiroydassarma (); balentskim (); Herbut1 (); lairoygoswami (); herbut1a (); Herbut2 (); kharitonov (); arago (); BJYang (); Fang2015 (); Yang16 (); brydon1 (); brydon2 (); Herbut3 (); savary1 (); roynevidomskyy (); savary2 (); sato (); CXLiu1 (); CXLiu2 (); brydon3 (); sunbinLee ().
The effects of electronic interactions on spin3/2 fermions are addressed within the framework of an extended Hubbardlike model, composed of only the shortrange components of repulsive Coulomb interactions. ^{1}^{1}1In this work we neglect the longrange tail of the Coulomb interaction. When the chemical potential is pinned at the band touching point, longrange Coulomb interaction can give rise to an infrared stable nonFermi liquid fixed point Abrikosov1 (); Abrikosov2 (); balentskim (), which, may however be unstable toward the formation of an excitonic phase kohn (); Herbut1 (); herbut1a (). By contrast, at finite chemical doping longrange interaction suffers conventional ThomasFermi screening and becomes shortranged in nature. A systematic incorporation of the longrange interaction into our discussion is left for a future investigation. Due to the vanishing density of states (DoS) in a Luttinger system (namely, ), sufficiently weak shortrange interactions are irrelevant perturbations. Therefore, any ordering takes place at finite coupling. We here employ a renormalization group (RG) analysis to construct various cuts of the global phase diagram at zero as well as finite chemical doping [see Figs. 3, 4 and 5], and combine it with meanfield analysis to gain insight into the organizing principle among distinct brokensymmetry phases (BSPs) ^{2}^{2}2All cuts of the global phase diagram are obtained from a RG analysis, accounting for interaction effects on Luttinger fermions. Such an RG analysis can only predict the phase boundaries between the Luttinger (semi)metal and various BSPs. However, our RG analysis is not applicable inside a BSP and cannot capture orderorder transitions. The color fillings in Figs. 3, 4, 5, 6, 7 and 16 are implemented solely for the clarity of presentation. . A gist of our findings can be summarized as follows.
1. By computing the reconstructed band structure (within the meanfield approximation) we organize dominant BSPs according to the condensation energy and entropy gains. Results are summarized in Fig. 2. We note that while the stiffness (uniform or anisotropic) of the band gap measures the condensation energy gain, the amount of gapless quasiparticles (resulting in powerlaw scaling of DoS at low energies) measures the entropy inside the ordered phase. All cuts of the global phase diagram (obtained from a RG analysis) show that the low (high) temperature phase yields larger condensation energy (entropy) gain, see Fig. 3.
2. The quasiparticle spectra inside the wave paired state and two nematic phases (belonging to the and representations of the cubic or point group) are fully gapped. Hence, nucleation of these three phases leads to the maximal gain of condensation energy, and they appear as the dominant BSPs at zero temperature, as shown in Fig. 3.
3. At finite temperature condensation energy gain competes with the entropy, and phases with higher entropy are realized at higher temperatures. Onset of any nematicity results in an anisotropic gap, in contrast to the situation with an wave pairing. Thus former orderings are endowed with larger (smaller) entropy (condensation energy), and are found at higher temperatures in Figs. 3, 3. By contrast, the dominant magnetic orders (belonging to the and representations) produce gapless Weyl quasiparticles and result in scaling of the DoS at low energies. Hence, these two magnetic orders carry larger entropy than the nematic phases or the wave pairing, and can only be found at finite temperature, see Figs. 3, 3. Luttinger semimetal (LSM) magnetic order (results from the allin allout state in pyrochlore lattice Savrasov (); Balents3 ()) transition at finite temperature is consistent with the experimental observation in NdIrO takagi (), while the magnetic ordering (yielding a 3in 1out ordering in pyrochlore lattice and supporting anomalous Hall effect goswamiroydassarma ()) can be germane for PrIrO AHE1 (); AHE2 (); AHE3 () ^{3}^{3}3ARPES measurements in NdIrO Exp:Nakatsuji2 () and PrIrO Exp:Nakatsuji1 () suggest that the LSM in these compound is isotropic..
4. Local four fermion interactions in the nematic channels are conducive for wave pairing (at zero and finite chemical doping), see Fig. 3, 3 and 4 , whereas shortrange magnetic interactions give birth to topological wave pairing (only at finite chemical doping), see Fig. 5. We further elaborate such an emergent “selection rule” in Sec. II.2. Confluence of magnetic order and wave pairing (resulting in gapless BdG quasiparticles, found in YPtBi Exp:Paglione2 ()) is in (qualitative) agreement with the global phase diagram of LnPdBi Exp:Paglione1 ().
The theoretical approach outlined in this work is quite general and can be extended to address the effects of electronic interactions in various strongly correlated multiband systems, among which twodimensional Dirac and quadratic fermions (respectively relevant for monolayer and bilayer graphene) grapheneRMP (); Balatsky (), threedimensional doped topological, crystalline and Kondo insulators TIRMP1 (); TIRMP2 (); TKIReview (), Weyl materials WeylRMP (), twisted bilayer graphene tblg1 (); tblg2 (); tblg3 (), are the most prominent and experimentally pertinent ones. In the future we will systematically study these systems.
i.1 Outline
The rest of the paper is organized as follows. In the next section we present an extended summary of our main results. The lowenergy description of the Luttinger model and its symmetry properties are discussed in Sec. III. In Sec. IV we discuss the reconstructed band structure inside various excitonic and superconducting phases. In Sec. V we introduce the interacting model for spin3/2 fermions and analyze the propensity toward the formation of various orderings within a meanfield approximation. Sec. VI is devoted to a renormalization group (RG) analysis of interacting Luttinger fermions at zero and finite temperatures and chemical doping. We summarize the main results and highlight some future outlooks in Sec. VII. Additional technical details are relegated to appendices.
Ii Extended Summary
Our starting point is a collection of spin3/2 fermions for which the normal state is described by a biquadractic touching of Kramers degenerate valence and conduction bands. The corresponding Hamiltonian operator is luttinger (); murakamizhangnagaosa ()
(1) 
where s are five wave harmonics in three dimensions, s are five mutually anticommuting four dimensional Hermitian matrices, and is the effective mass for gapless excitations in the orbitals in a cubic environment. Momentum and chemical potential are measured from the band touching point. Additional details of this model are discussed in Sec. III and Appendix C. The mass anisotropy parameter () lies within the range goswamiroydassarma (). But, for the sake of concreteness we restrict our focus on the isotropic system with . For discussion on the evolution of phase diagrams with varying , see Secs. V.1 and VI, and Figs. 6 and 16. The LSM is realized as an unstable fixed point at , the red dot in Fig. 1. This fixed point is characterized by the dynamic scaling exponent , determining the relative scaling between energy and momentum according to . The chemical potential is a relevant perturbation at this fixed point, with the scaling dimension . Hence, the correlation length exponent at this fixed point is . Therefore, the LSM can be envisioned as a quantum critical point (QCP) separating electrondoped (for , the brown region) and holedoped (for , the green region) Fermi liquid phases, as shown in Fig. 1. Our discussion is focused on the quantum critical regime (the shaded region).
The crossover temperature () separating the quantum critical regime accommodating gapless spin3/2 excitations from the Fermi liquid phases can be estimated in the following way
(2) 
where is a characteristic length scale and is the carrier density. Two critical exponents ( and ) and the dimensionality of the system () determine the scaling of various thermodynamic (such as specific heat, compressibility) and transport (such as dynamic conductivity) quantities in this regime.
ii.1 Critical scaling in noninteracting system
The freeenergy density (up to an unimportant temperature () independent constant) inside the critical regime is given by (setting )
(3) 
where represents the polylogarithimic function. The specific heat of this system is given by
(4) 
for (ensuring that the system resides inside the critical regime), where and . From the above expression of the freeenergy density we can also extract the scaling of compressibility (), given by
(5) 
where . Therefore, the presence of finite chemical doping does not alter the leading powerlaw scaling of physical observables, such as , , but only provides subleading corrections, which are suppressed by a parametrically small quantity . ^{4}^{4}4The scaling of specific heat and compressibility is determined by the dimensionality () and dynamic scaling exponent () according to and , respectively. Also note that
(6) 
is a universal ratio, capturing the signature of a quantum critical point in . A detailed derivation of this analysis is shown in Appendix A. Qualitatively similar subleading corrections are also found in the scaling of the dynamic conductivity, which we discuss now.
Gauge invariance mandates that the conductivity () must scale as . Hence for a collection of quasiparticles (such as the Luttinger fermions), or in three spatial dimensions. Indeed we find that the Drude (Dr) component of the dynamic conductivity in the Luttinger system is given by [see Appendix B]
(7) 
where is a monotonically increasing universal function of its argument [see Eq. (70) and Fig. 21] and is the frequency. On the other hand, the interband (IB) component of the optical conductivity reads as
(8) 
Hence, interband component of the optical conductivity vanishes as as and the LSM can be identified as a powerlaw insulator.
Therefore, even when the chemical doping is finite there exists a wide quantum critical regime, shown in Fig. 1, where the scaling of thermodynamic and transport quantities are essentially governed by quasiparticles, and the chemical potential provides only subleading corrections. Next we highlight the imprint of finite temperature and chemical doping on the global phase diagram of interacting spin3/2 fermions.
ii.2 Electronelectron interactions in a Luttinger (semi)metal
In this work we compute the effects of electronelectron interactions on Luttinger fermions, occupying the critical regime of the noninteracting fixed point, see Fig. 1. In this regime any shortrange or local fourfermion interaction () is an irrelevant perturbation, since
due to the vanishing DoS, namely . We use a RG analysis, tailored to address the effects of electronic interactions on Luttinger fermions in , constituting a band structure, to arrive at various cuts of the global phase diagram of this system. If, on the other hand, temperature and chemical doping are such that the system resides inside a Fermi liquid phase, the notion of nodal quasiparticles becomes moot and our RG analysis loses its jurisdiction. ^{5}^{5}5 Note that in the presence of a Fermi surface the DoS is constant and the interaction coupling is dimensionless. Consequently a Fermi liquid becomes unstable toward the formation of a superconductor (often nonwave) even in the presence of repulsive electronelectron interactions, following the spirit of the KohnLuttinger mechanism SankarRGRMP (); kohnLuttinger (); KLChubukovReview (); zanchischulz (); hlubina (); raghukivelson () and the superconducting transition temperature () mimics the BCSscaling law . Furthermore, we augment the RG analysis with an organizing principle based on the competition between energy and entropy. To this end we rely on the computation of the reconstructed band structure inside the ordered phases within the meanfield approximation. Subsequently, we also promote a “selection rule” among neighboring phases in the global phase diagram, originating purely from their algebraic or symmetry properties. For the sake of simplicity we concentrate on the isotropic system () in the following three subsections. Nonetheless, our results hold (at least qualitatively) for any arbitrary value of , as summarized in the last subsection.
IIB1. Organizing principle: Emergent Topology & EnergyEntropy
Let us first promote an organizing principle among BSPs according to their contribution to the energy and entropy gain and anticipate their presence in the global phase diagram. In what follows we highlight the reconstructed band structure inside the dominant ordered phases within a meanfield or HartreeFock approximation, which by construction undermines the ordered parameter fluctuations. The emergent band topology is computed by diagonalizing an effective singleparticle Hamiltonian, composed of the noninteracting Luttinger Hamiltonian and corresponding order parameter, see Sec. IV.3 for details.
Perhaps it is natural to anticipate that at zero temperature strong electronic interactions favor the phases that produce the largest spectral gap, as the onset of these ordered states offers maximal gain of condensation energy. In a LSM there are three candidate BSPs that yield fully gapped quasiparticle spectra: (a) an wave superconductor, producing a uniform and isotropic gap, and (b) two nematic orders (belonging to the and representations), producing anisotropic gaps. As shown in Fig. 3, only these three phases can be found in an isotropic and interacting LSM at zero temperature.
The energyentropy competition, leading to an organizing principle among competing phases at finite temperature, can be appreciated from the scaling of the DoS or the stiffness (uniform or anisotropic) of the spectral gap. As mentioned above, the wave pairing and nematic orders respectively produce uniform and anisotropic gaps, whereas two magnetic orders, belonging to the and representations, respectively produce eight Balents3 () and two goswamiroydassarma () isolated simple Weyl nodes, around which the DoS vanishes as for sufficienlty low energies. On the other hand, each copy of the wave pairings accommodates two nodal loops for which the lowenergy DoS scales as [see Sec. IV.4] brydon1 (); Herbut3 (); roynevidomskyy (); savary2 (); sunbinLee (). Since we are interested in energy or temperature scales much smaller than the ultraviolet cutoff or bandwidth (), the structure of the spectral gap (isotropic or anisotropic) and powerlaw scaling of DoS carry sufficient information to organize the ordered phases according to their contribution to condensation energy and entropy, summarized in Fig. 2. In brief, existence of more gapless points (resulting in higher DoS near ) yields larger entropy, while a more uniform gap leads to higher gain in condensation energy. From various cuts of the global phase diagram at finite temperature, see Fig. 3, we note the following common feature: Among competing orders, the one with maximal gain in condensation energy appears at low temperature, while the phase with higher entropy is realized at higher temperature, in accordance with the general principle of energyentropy competition. Since the DoS in a LSM scales as (maximal entropy), it can always be found at sufficiently high (weak) enough temperature (interactions).
A similar conclusion can also be arrived at when the chemical potential is placed away from the band touching point. At finite chemical doping all particlehole (two nematic and two magnetic) orders produce a Fermi surface (according to the Luttinger theorem Luttinger_Theorem ()) and hence a finite DoS. By contrast, any superconducting order at finite doping maximally gaps the Fermi surface. Therefore, at finite chemical doping superconducting orders are energetically superior to the excitonic orders, and they can be realized at sufficiently low temperature even in the presence of repulsive electronic interactions. Concomitantly, the particlehole orders are pushed to the higher temperature and interaction regime, see Figs. 4 and 5.
Even though we gain valuable insights into the organization of various BSPs in the global phase diagram of strongly interacting spin3/2 fermions from the competition between energy and entropy inside the ordered phases (guided by emergent topology of reconstructed band structure), the phase diagrams shown in Figs. 3, 4 and 5 are obtained from an unbiased RG analysis, which systematically accounts for quantum fluctuations beyond the saddle point or meanfield approximation. Next we highlight the key aspects of the RG analysis.
IIB2. Methodology: Renormalization Group
The RG analysis we pursue in this work is controlled by a “small” parameter , measuring the deviation from the lower critical two spatial dimensions () of the theory, where local fourfermion interactions are marginal, with , and hence . Both temperature and chemical potential (bearing the scaling dimension of energy) are relevant perturbations at the fixed point, with scaling dimension . The leading order RG analysis can be summarized in terms of the following set of coupled flow equations
(9) 
where is the logarithm of the RG scale, and are respectively the dimensionless temperature and chemical potential. For brevity we take in the above flow equations. Here represents the set of dimensionless fourfermion interactions with and are functions of the mass anisotropy parameter , and . The RG flow equations for s are obtained by systematically accounting for quantum corrections to the quadratic order in the s. The relevant Feynman diagrams are shown in Fig. 14. A more detailed discussion of the RG analysis is presented in Sec. VI and Appendix E. Some salient features of the RG analysis are the followings.

Temperature () and chemical potential () provide two infrared cutoffs chakravarty (); vafek (), respectively given by
(10) for the flow of quartic couplings , where and represent the bare values (). Ultimately stops the flow of fourfermion interactions. At zero temperature and chemical doping the system is devoid of any such natural infrared cutoff, implying .

Any weak local fourfermion interaction is an irrelevant perturbation and all orderings (realized when ) take place at finite coupling through quantum phase transitions (QPTs). Such QPTs are controlled by quantum critical points (QCPs) and all transitions are continuous in nature. The universality class of the transition is determined by two critical exponents, given by
(11) and for the physically relevant situation .
Using the RG analysis we arrive at various cuts of the global phase diagram of interacting spin3/2 fermions at (1) zero chemical doping [see Fig. 3] and (2) for finite [see Figs. 4 and 5]. The universality class of the QPT leaves its signature on the scaling of the transition temperature (). Note that sondhiRMP (); sachdevbook (), where is the reduced distance from the critical point, located at . Hence, for and , obtained from the leading order expansion, after setting , irrespective of the choice of the coupling constant and the resulting BSP (see Fig. 17). We discuss this issue in detail in Sec. VI.3.
IIB3. Competing Orders & Selection Rule
The correspondence between a given interaction coupling and the resulting phases can be appreciated by formulating the whole theory in the basis of an eight component Nambudoubled spinor , introduced in Sec. III.3. In this basis the Luttinger Hamiltonian . Pauli matrices operate on the Nambu or particlehole index. Any fourfermion interaction takes the form and an order parameter () couples to a fermion bilinear according to , where and are eight dimensional Hermitian matrices. We argue that when is sufficiently strong, it can support only two types of ordered phases, for which ^{6}^{6}6If and are multicomponent vectors of and elements, respectively, then condition (2) is satisfied when at least matrices anticommute with matrices, where is the ceiling function.
(12) 
This outcome can be appreciated in the following way.
When an interaction coupling diverges toward under coarse graining (indicating onset of a BSP), it provides positive scaling dimension to an order parameter field only when one of the above two conditions is satisfied. We substantiate this argument by considering the relevant Feynman diagrams [see Fig. 15] in Sec. VI.5. Even though we arrive at this “selection rule” among competing orders from a leading order RG calculation, such a simple argument relies on internal symmetries among competing orders (breaking different symmetries) and is expected to hold at the nonperturbative level. We now support this claim by focusing on a specific example.
Let us choose a particular fourfermion interaction (in the nematic channel)
From the phase diagrams shown in Figs. 3 for zero and finite temperature and 4(left) for finite chemical doping, we find that when this coupling constant is sufficiently strong, it supports two distinct BSPs.

A nematic order following the representation, for which . In this case selection rule (1) from Eq. (12) is satisfied, since , and hence .

An wave superconductor following the trivial representation, for which , where is a fourdimensional identity matrix. The onset of wave pairing follows from selection rule (2) in Eq. (12), since .
Moreover we realize that the nematic order and the wave pairing together constitute an vector, , of five mutually anticommuting matrices, reflecting the enlarged internal symmetry between these two orders. Following the same spirit we arrive at the following observations.

Four fermion interaction in the nematic channel () supports (a) a nematic order transforming under the representation [satisfying selection rule (1)] and (b) an wave superconductor [satisfying selection rule (2)], as shown in Figs. 3 and 4(right). One can construct an vector by combining these two orderparameters [see Eq. (43)].

Four fermion interaction in the channel () supports (a) a magnetic order transforming under the representation [satisfying selection rule (1)], and (b) nematic order and wave pairings [satisfying selection rule (2)], as shown in Figs. 3 and 5(top). Notice, we can construct multiple copies of composite SU(2) order parameters, by combining the order with nematic or wave pairing, see Figs. 18(a) and 19.

Four fermion interaction in the magnetic channel () supports (a) a magnetic order transforming under the representation [satisfying selection rule (1)], and (b) nematic and wave pairings [satisfying selection rule (2)], as shown in Figs. 3 and 5(bottom). Combining the magnetic order with nematic or wave pairing we can construct multiple copies of composite SU(2) vector, see Figs. 18(b), 18(c) and 20.
A more detailed discussion supporting these scenarios is presented in Sec. VI.5. Therefore, combining the energyentropy competition (obtained from the reconstructed band topology within a meanfield approximation) and an unbiased (controlled by a small parameter ) RG analysis with the selection rule we gain valuable insights into the nature of broken symmetry phases, competing orders and quantum critical phenomena in the global phase diagram of strongly interacting spin3/2 fermions.
IIB4. Anisotropic Luttinger (semi)metal
So far, we centered our focus on the isotropic system [realized for in Eq. (1)]. Note that for the system enjoys an enlarged (but artificial) SO(3) rotational symmetry. However, in a cubic environment in general. Nonetheless, all the central results we quoted in the last three subsections hold (at least qualitatively) for any arbitrary value of : . The discussion on the role of the mass anisotropy parameter on the global phase diagram of interacting spin3/2 fermions is rather technical, which we address in depth in Secs. V.1, VI.1, VI.4. We here only quote some key results, which nicely corroborate with the rest of the discussion from this section.

We identify the mass anisotropy parameter as a valuable nonthermal tuning parameter, and for suitable choices of this parameter one can find (a) nematic and magnetic order respectively for strong enough and couplings (when ), see Figs. 6 and 6, (b) nematic and magnetic orders for strong enough and (when ), as shown in Figs. 6 and 7 at zero temperature and chemical doping, respectively. These outcomes are in agreement with selection rule (1).

At finite chemical doping (a) an wave pairing emerges from repulsive electronic interaction in the channel [see Fig. 6] as well as channel [see Fig. 6], (b) wave pairings belonging to the and representations respectively appear for repulsive interaction in the channel [see Fig. 6] and channel [see Fig. 6]. These outcomes are in accordance with selection rule (2), as we argued previously for an isotropic Luttinger system.
We now proceed to a detailed discussion on each component of our work, starting from the noninteracting Luttinger model.
Iii Luttinger model
We begin the discussion with the Luttinger model describing a biquadratic touching of Kramers degenerate valence and conduction bands at an isolated point (here chosen to be the point, for convenience) in the Brillouin zone. In this section, we first present the lowenergy Hamiltonian and discuss its symmetry properties (Sec. III.1). Subsequently, we introduce the corresponding imaginary time () or Euclidean action and the notion of the renormalization group (RG) scaling (Sec. III.2). Finally, we define an eightcomponent Nambudoubled basis that allows us to capture all, including both particlehole or excitonic and particleparticle or superconducting, orders within a unified framework (Sec. III.3).
iii.1 Hamiltonian and Symmetries
The Hamiltonian operator describing a biquadratic touching of Kramers degenerate valence and conduction bands in three dimensions is given by luttinger (); murakamizhangnagaosa ()
(13) 
where is the fourdimensional identity matrix. Chemical potential and momenta are measured from the band touching point. Here, is a fivedimensional unit vector that transforms under the representation under the orbital rotations. Hence is constructed from the wave form factors or spherical harmonics , as shown in Appendix C. The corresponding fourcomponent spinor basis is given by
(14) 
where is the fermionic annihilation operator with momenta and spin projection and . The five mutually anticommuting matrices are defined as
(15) 
Two sets of two dimensional Pauli matrices and respectively operate on the sign () and magnitude () of the spin projections, where . To close the Clifford algebra of all fourdimensional Hermitian matrices we also define ten commutators according to , with and . All sixteen fourdimensional matrices can be expressed in terms of the products of spin3/2 matrices (), as also shown in Appendix C.
The energy spectra for Luttinger fermions are given by , where for
(16) 
reflecting the quadratic band touching for , which is protected by the cubic symmetry. For brevity we drop the explicit dependence of on .
Notice that the independence of on manifests the Kramers degeneracy of the valence and conduction bands, ensured by (1) the timereversal () and (2) the parity or inversion () symmetries. Specifically, under the reversal of time, and and hence , where is the complex conjugation, yielding (reflecting Kramers degeneracy of bands). Under the inversion and .
The “average” mass and the mass anisotropy parameter are respectively given by goswamiroydassarma ()
(17) 
Note that for and belong to the (three component) and (two component) representations of the cubic or octahedral () point group, and and are effective masses in these two orbitals, respectively. The mass anisotropy parameter allows us to smoothly interpolate between (1) the limit when the dispersion of the orbital becomes flat, yielding and (2) when the orbital becomes nondispersive, leading to . For or , the Luttinger model enjoys an enlarged spherical symmetry. Any captures a quadrupolar distortion in the system (still preserving the cubic symmetry). In what follows, we treat as a nonthermal tuning parameter to explore the territory of interacting Luttinger fermions.
The connection between the spin projections ( and ) and the bands can be appreciated most economically by taking . For such a specific choice of momentum axis, the Luttinger Hamiltonian takes a diagonal form, given by
(18) 
From the above expression we can immediately infer that the pseudospin projections on the valence and conduction bands are respectively and .
iii.2 Lagrangian and Scaling
The imaginary time () Euclidean action corresponding to the noninteracting Luttinger model is given by
(19) 
The action remains invariant under the following rescaling of space(imaginary)time coordinates and the fermionic field
(20) 
where is the dynamic scaling exponent, measuring the relative scaling between energy and momentum according to . For Luttinger fermions . The parameter is the logarithm of the RG scale. In what follows in Secs. V and VI, we use the above scaling ansatz while addressing the effects of electronic interactions in this system. Under the above rescaling of parameters, the temperature () and chemical potential () scale as
(21) 
Therefore, the scaling dimension of these two quantities is (same as that of energy). Throughout, we use the natural unit, in which .
iii.3 Nambu doubling
To facilitate the forthcoming discussion we here introduce an eightcomponent Nambudoubled spinor basis (suitable to capture both exitonic and superconducting orders within a unified framework) according to
(22) 
where is a fourcomponent spinor, see Eq. (14). In the lower block of we absorb the unitary part of the timereversal operator , ensuring that the eightcomponent Nambu spinor () transforms the same way as the original four component spinor under the pseudospin rotation. In this basis the eightdimensional Luttinger Hamiltonian takes a simple form
(23) 
and the timereversal operator becomes . The newly introduced set of Pauli matrices operates on the Nambu or particlehole indices, with . Therefore, by construction while the excitonic orders assume blockdiagonal form, all superconducting orders are blockoffdiagonal in the Nambu subspace. Note that commutes with the number operator .
Iv Broken symmetry phases
Next we discuss possible BSPs in this system. We introduce various possible excitonic and superconducting orders in the Nambu basis () in two subsequent sections. Finally, we discuss the reconstructed band structure and emergent topology inside the ordered phases.
iv.1 Particlehole or excitonic orders
The effective singleparticle Hamiltonian in the presence of all possible momentumindependent or local or intraunit cell excitonic orders is given by
(24) 
where
The ordered phases can be classified according to their transformation under the cubic () point group symmetry. Regular fermionic density () does not break any symmetry (hence does not correspond to any ordering) and transforms under the trivial representation. A threecomponent nematic orderparameter, constituted by , transforms under the representation. By contrast, a twocomponent nematic order transforming under the representation is captured by . Both of them break only the cubic symmetry, but preserve timereversal and inversion symmetries. The ordered phase represents either a timereversal invariant insulator or a Dirac semimetal, about which more in a moment [see Sec. IV.3]. Since five matrices transform as components of a rank2 tensor under SO(3) rotations, the two nematic phases represent quadrupolar orders, see Appendix C.
All ordered phases shown in the second line of Eq. (IV.1) break timereversal symmetry and represent different magnetic orders. For example, corresponds to an octupolar order (since ), transforming under the singlet representation. In a pyrochlore lattice of 227 iridates such an ordered phase represents the “allin allout” arrangement of electronic spin between two adjacent cornershared tetrahedra Savrasov (); Balents3 (). By contrast, “twoin twoout” or “spinice” magnetic orderings on a pyrochlore lattice are represented by a threecomponent vector (accounting for six possible twoin twoout arrangements in a single tetrahedron). Since such an ordered phase contains a linear superposition of dipolar and octupolar moments, and transforms under the representation goswamiroydassarma (). Any other magnetic ordering can be represented by a six component vector with and . No physical realization of such multicomponent magnetic ordering in any material is currently known, and we do not delve into the discussion on such ordering for the rest of the paper.
iv.2 Particleparticle or superconducting orders
The effective single particle Hamiltonian in the presence of all possible momentumindependent or local or intraunit cell superconducting orders reads brydon1 (); Herbut3 (); savary1 (); roynevidomskyy ()
(26) 
where
(27)  
and is the global superconducting phase. Any pairing proportional to preserves (breaks) timereversal symmetry (recall that the timereversal operator in the Nambu basis is ). Here, is the amplitude of the wave pairing, transforming under the representation. The wave pairing breaks only the global symmetry, but preserves the cubic symmetry. On the other hand, captures the amplitude of three wave pairings (for ) transforming under the representation, and for represents the amplitude of two wave pairings belonging to the representation. Notice can be expressed in terms of the product of two spin3/2 matrices, and all five wave pairings break the cubic symmetry, while introducing a lattice distortion or electronic nematicity in the system. Hence, they stand as representatives of quadrupolar nematic superconductors.
iv.3 Reconstructed band structure and emergent topology
Next we consider the reconstructed band structure inside different BSPs which provides valuable information regarding the emergent topology inside ordered phases. The onset of any ordering discussed in the previous sections destabilizes the biquadratic touching and gives rise to either gapped or gapless quasiparticles (see below). Furthermore, this exercise will allow us to appreciate the energyentropy competition among different orderings [see Sec. IV.4], which ultimately plays a decisive role in the organization of various phases in the global phase diagram of interacting Luttinger fermions.
1. nematicity: The three component orderparameter for the nematic phase gives birth to gapless quasiparticles for the following four configurations
These four phase lockings are respectively shown as blue, red, green and black points in Fig. 8. The gapless phase corresponds to a topological Dirac semimetal (since nematicity preserves the Kramers degeneracy of valence and conduction bands), similar to the ones recently found in CdAs cdas:Exp () and NaBi nabi:Exp (). The DoS in a Dirac semimetal vanishes as . The Dirac points are located along the body diagonals (the axes) of a cubic system and respectively placed at
where , as shown in Fig. 8. For any other phase locking within the sector the system becomes an insulator. The spectral gap in the insulating phase is anisotropic and it is energetically superior over the gapless Dirac semimetal phase.
2. nematicity: The two component nematic order is most conveniently described in terms of the following parametrization
(30) 
where is the internal angle in the orderparameter space. Only for
(31) 
the quasiparticle spectra are gapless, as shown in Fig. 9, and the ordered phase represents a topological Dirac semimetal. Specifically, for , the Dirac points are respectively located on , and axes (the axes), see Fig. 9, and the separation of two Dirac points is given by , where . For any other phase locking within the sector, the system becomes an insulator.
3. magnet: In the presence of an octupolar ordering, the twofold degeneracy of the valence and conduction band gets lifted and a pair of Kramers nondegenerate bands touch each other at the following eight points in the Brillouin zone [see Fig. 10]
(32) 
where . They represent simple Weyl points, which act as source (4 of them) and sink (4 of them) of Abelian Berry curvature of unit strength. However, due to an octupolar arrangement of the Weyl nodes, the net Berry curvature through any highsymmetry plane is precisely zero and this phase does not support any anomalous Hall effect. The DoS at low energies then scales as Savrasov (); Balents3 (); goswamiroydassarma ().
4. magnet: For each component of magnetic order (represented by the matrix operator with ) the ordered phase supports two Weyl nodes along one of the axes and a nodalloop in the corresponding basal plane. For example, when the left and right chiral Weyl nodes are located at where and a nodalloop is found in the plane, as shown in Fig. 10. Similarly, for and the Weyl nodes are separated along the and axes, and the nodalloops are respectively found in the and planes. Due to the presence of two Weyl nodes, each configuration of twoin twoout magnetic order supports a finite anomalous Hall effect in the plane perpendicular to the separation of the Weyl nodes. However, any triplet magnetic order, represented by , gets rid of the nodal loop and supports only two Weyl nodes along one of the bodydiagonals ( axes). Hence, triplet magnetic orders are energetically favored over their uniaxial counterparts goswamiroydassarma (). ^{7}^{7}7The low energy DoS in the presence of a nodal loop and two point nodes (due to a uniaxial order) is dominated by the former and scales as , while in a triplet state the DoS scales as (due to the point nodes). Hence, formation of the triplet ordering causes powerlaw suppression of the DoS and increases the condensation energy gain.
5. or wave pairing: Notice that the matrix operator representing an wave pairing fully anticommutes with the Luttinger Hamiltonian (for any value of ) and thus corresponds to a mass for Luttinger fermions. The quasiparticle spectra inside the paired state is fully gapped, but the phase is topologically trivial.
6. pairing: Three wave pairings, proportional to , and matrices, belong to the representation and respectively possess the symmetry of , and pairings. Each component supports two nodal loops in the ordered phase, as shown in the first three rows of Table 1 brydon1 (); roynevidomskyy (). Two nodal loops for the or pairing are shown in Fig. 11. The two nodal loops for or and or pairings can respectively be obtained by rotating the ones shown for pairing by , with respect to the and axes.
7. pairing: pairings proportional to and matrices respectively possess the symmetry of and pairings and each of them supports two nodal loops, as shown in the last two rows of Table 1 brydon1 (); roynevidomskyy (). Note that two nodal loops for the pairing can be realized by rotating the ones for the pairing by about the axis. However, two nodal loops for the pairing, shown in Fig. 11, cannot be rotated into the ones for pairing. Therefore, despite the fact that the and pairings belong to the same representation, they are not energetically degenerate roynevidomskyy (); sigristRMP (). Since the radius of the nodal loops for the pairing is the smallest, this paired state is the energetically most favorable among five wave pairings. ^{8}^{8}8Even though type, such as , pairing can eliminate nodal loops from the spectra in favor of point nodes around which , strong interband coupling causes inflation of such nodes and yields Fermi surface of BdG quasiparticles, leading to a constant DoS at lowest energy, followed by brydon2 (). Presently, it is not very clear between (a) individual wave pairing and (b) type pairing, which one is energetically more advantageous.
Pairing  IREP.  Equations for nodal loops  Symmetry 

,  
,  
, 