Bose-Einstein condensation in a frustrated triangular optical lattice

# Bose-Einstein condensation in a frustrated triangular optical lattice

Peter Janzen, Wen-Min Huang111Email:wenmin@phys.nchu.edu.tw, L. Mathey222Email:lmathey@physnet.uni-hamburg.de Zentrum für Optische Quantentechnologien and Institut für Laserphysik, Universität Hamburg, 22761 Hamburg, Germany
Department of Physics, National Chung-Hsing University, Taichung 40227, Taiwan
The Hamburg Centre for Ultrafast Imaging, Luruper Chaussee 149, Hamburg 22761, Germany
July 1, 2019
###### Abstract

The recent experimental condensation of ultracold atoms in a triangular optical lattice with a negative effective tunneling parameter paves the way to study frustrated systems in a controlled environment. Here, we explore the critical behavior of the chiral phase transition in such a frustrated lattice in three dimensions. We represent the low-energy action of the lattice system as a two-component Bose gas corresponding to the two minima of the dispersion. The contact repulsion between the bosons separates into intra- and inter-component interactions, referred to as and , respectively. We first employ a Huang-Yang-Luttinger approximation of the free energy. For , which corresponds to the bare interaction, this approach suggests a first order phase transition, at which both the U symmetry of condensation and the symmetry of the emergent chiral order are broken simultaneously. Furthermore, we perform a renormalization group calculation at one-loop order. We demonstrate that the coupling regime shares the critical behavior of the Heisenberg fixed point at . For we show that flows to a negative value, while increases and remains positive. This results in a breakdown of the effective quartic field theory due to a cubic anisotropy, and again suggests a discontinuous phase transition.

###### pacs:
67.85.Hj, 03.75.Mn, 64.60.ae, 75.10.Hk

## I Introduction

The seminal studies Anderson95 (); Davis95 (); Bradley95 () reported the observation of Bose-Einstein condensation (BEC) in time-of-flight images, which revealed that ultracold bosons accumulate near the minimum of the dispersion relation and condense below a critical temperature. This second order phase transition is the quintessential example for U symmetry breaking, described by complex theory. Subsequently, numerous studies have addressed the question if and how BEC can be achieved, by including symmetries apart from the conventional U symmetry Lewenstein07 (); Bloch08 (). Populating bosons in spatially anisotropic orbitals of an optical lattice, for instance, has been proposed to create exotic superfluid orders Browaeys05 (); Isacsson05 (); Liu06 (); Kuklov06 (); muller07 (); Lim08 (); Sarma08 (); Wu09 (); Sarma11 (); Li11 (); Cai11 (); Hemmerich11 (); Panahi12 (); Liu12 (). In the experiment reported in Wirth11 (); Lewenstein11 (), a long-lived metastable state of ultracold bosons in the -band was indeed realized and displays condensation involving two different momenta, visible in time-of-flight measurements. Both experimental work Hemmerich13 () as well as theoretical analysis Liu06 (); Wu09 () suggest that the two momentum states form a superposition with an imaginary relative phase, in which the atoms condense, leading to superfluid order with broken time-reversal symmetry. By exploiting a matter wave heterodyning technique, the breaking of time-reversal symmetry via the spontaneous formation of chiral order in this -band bipartite optical lattice is revealed without ambiguity Kock14 ().

Another method to create unconventional BEC is to load bosons into the Floquet states of a shaken optical lattice Eckardt05 (); Gemelke (); Lignier07 (); Zenesini09 (); Chin13 (); Chin15 (). In such a shaken triangular lattice the effective tunneling parameters can be tuned to negative values Lewenstein10 (); Struck11 (). The bosonic atoms condense at the minima of the effective dispersion at non-zero momenta. This was used to perform simulations of frustrated classical magnetism in Struck11 (). Furthermore, complex-valued effective tunneling parameters can be created, which correspond to artificial gauge fields Garcia12 (); Struck12 (). These artificial gauge fields couple to an emergent, chiral order parameter, as demonstrated in Ref.Struck13 ().

The frustrated bosonic system studied in Ref.Struck13 (), as sketched in Fig. 1 (a), is related to a classical XY model in the triangular lattice by ignoring phase fluctuations along the z-axis. The underlying physics is to represent the bosonic fields in the phase-density representation, thus the phase is mapped onto a classical XY spin, Wen (). For a negative tunneling parameter in this approximation results in antiferromagnetic spin coupling between neighboring sites Struck13 (); Becker10 (). This two-dimensional (2D) antiferromagnetic XY model is frustrated due to its triangular geometry, and has two degenerate ground states, discriminated by their chiral order, as illustrated in Fig. 1 (c), Choi (); Yosefin (); Hasenbusch05 (). Thus, in addition to breaking the U symmetry of the system, spontaneous breaking of an emergent, chiral symmetry occurs Yosefin (); Hasenbusch05 (); Altman14 ().

In this paper, we explore the critical behavior of an interacting Bose gas, realized in Ref.Becker10 (); Struck11 (); Struck13 (). In the -plane, as shown in Fig. 1 (a), the bosons move in a triangular optical lattice with a real-valued, the negative effective tunneling parameter , while moving freely along the -direction. We therefore include the three dimensional character of the system, and the density fluctuations. We develop a field theoretical description of the system, which has the form of a two-component Bose gas. The two species describe the bosons near the two minima of the dispersion, as shown in Fig. 1 (b). In this description, the U symmetry is broken if one of the two species is condensed to the nearby minima. Moreover, the symmetry breaking is analogous to condensation at only one of the two degenerate energetic minima Struck13 (); Becker10 (). The weak contact repulsion of the bosons is also decomposed into the interaction within one component, , and the interaction between these two components, , in the two-component Bose gas.

By regarding these interactions as two independent parameters in a Huang-Yang-Luttinger (HYL) approach Huang57 (), we find that the critical behavior of the symmetry breaking is controlled by the ratio of these two interactions in the mean-field scheme. The HYL approximation predicts that the symmetry breaking is a continuous, second-order phase transition for . However, the symmetry breaking becomes discontinuous, first-order, for . For , the two (1) symmetries are spontaneously broken as the temperature is decreased, while the symmetry is preserved.

To further study the phase diagram beyond the mean-field scheme, we perform a one-loop renormalization group (RG) calculation, resulting in a flow of the chemical potential and the coupling constants. This calculation clarifies that there are two regimes: For , and flow towards a Heisenberg fixed point, at which , describing a second order phase transition that differs from the condensation transition of a single-component system. For , however, flows towards a negative value, while increases and remains positive. This result implies the breakdown of the quartic theory, and higher-order terms, for instance a three-body interaction, need to be included. It also indicates that a discontinuous phase transition occurs Domany (), due to a cubic anisotropy. Thus, we conclude that the critical behavior in the regime is of first order, improving on the HYL approximation. We conclude that for the bare interaction of the system, the ratio , suggests that the transition is first order.

This paper is organized as follows: in Sec. II we develop the field theoretic description of the lattice model. In Sec. III we study the free energy based on the Huang-Yang-Luttinger approximaiton and investigate the phase diagram for different ratios of . In Sec. IV, we study the critical behavior in the framework of a one-loop RG calculation, and in Sec. V we conclude.

## Ii Effective field theory

The system, as depicted in Fig. 1 (a), is described by the Hamiltonian , with

 H0=∫dz\leavevmode\nobreak {∑rψ†(r,z)(−ℏ2∂2z2m0−μ3D)ψ(r,z) +|Je|∑⟨r,r′⟩[ψ†(r,z)ψ(r′,z)+h.c.]}, (1)

and the interaction term

 HI=U2∑r∫dz\leavevmode\nobreak ψ†(r,z)ψ†(r,z)ψ(r,z)ψ(r,z). (2)

represents nearest-neighbor pairs of sites and is the mass of the atoms. is the chemical potential, and denotes the magnitude of the repulsive contact interaction. The frustration of the system is due to the negative value of the tunneling parameter . Neglecting the interaction term and the chemical potential, we obtain the dispersion relation

 ε(k)=|Je|[2cos(dλky)+2cos(dλ√3kx/2−dλky/2) +2cos(dλ√3kx/2+dλky/2)]+ℏ2k2z/(2m0), (3)

with the lattice spacing , and being the wavelength of the laser creating the 2D lattice potential Struck13 (). The dispersion is shown in Fig. 1 (b). It displays two energetically degenerate minima, located at the two distinct momenta , at the boundary of the first Brillouin zone. The classical spins depicted in Fig. 1 (c) correspond to the real and imaginary part of the plane waves and .

The hidden symmetry of the system emerges as follows. At low temperatures, the bosons will accumulate near the minima. If the bosons condense at one of the two minima, one symmetry is broken due to condensation, and the symmetry, which corresponds to the density imbalance between the minima. However, if the bosons condense at both minima with equal density, the symmetry is preserved, while both symmetries are broken.

We now derive the low-energy field theoretical description. At low temperatures, bosons will be distributed around the minima and . Therefore, we decompose the bosonic field in momentum space as

 ψ(r,z) = 1√N∑keik⋅rψ(k,z) (4) ≃ A2λ√N∑j=1,2eikj⋅r∫|qj|<Λqd2q4π2\leavevmode\nobreak eiqj⋅rϕj(qj,z) = Aλ∑j=1,2ϕj(r,z)\leavevmode\nobreak eikj⋅r,

where is the total number of sites in the xy-plane, and . We define the slowly varying fields via with , where is the momentum cutoff, and is the area in the xy-plane that corresponds to a single site, when mapped on a continuum description. By expressing the Hamiltonian of Eq. (II) in terms of the fields , we obtain

 Heff0=∫d3R∑j=1,2∑aϕ†j(r,z){−ℏ2∂2a2ma−μ}ϕj(r,z). (5)

Here, , , , , and is the chemical potential of the Bose gas in the continuum description. The fields have a quadratic dispersion relation centered at momentum , respectively,

 εeff(qj,kz)=ℏ2q2jx2mJ+ℏ2q2jy2mJ+ℏ2k2z2mz, (6)

with and . Expressed in terms of , the interaction in Eq. (2) is

 HeffI=∫d3R\leavevmode\nobreak {V02∑j=1,2ϕ†j(R)ϕ†j(R)ϕj(R)ϕj(R) +V12\leavevmode\nobreak ϕ†1(R)ϕ†2(R)ϕ2(R)ϕ1(R)}, (7)

where is the intra-component interaction and is the inter-component one. It is noticed that the effective interaction is forbidden to exchange species because of the momentum conservation of the original Hamiltonian, Eq. (2). Secondly, the bare magnitudes of these parameters are and , so in particular we have . However, under the RG flow that we derive below, these parameters will flow independently. This forces us to regard and as two independent variables throughout, because their ratio is not protected by a symmetry. Furthermore, our analysis covers the full universality class of this type, not necessarily limited to the original system on a triangular lattice. The total effective Hamiltonian is a two-component complex -theory, with a single chemical potential , because only the total density of the bosons is conserved.

## Iii Huang-Yang-Luttinger approach

As a first insight into the critical behavior of the system, we compute the free energy of the effective Hamiltonian within the Huang-Yang-Luttinger approximation Huang57 (). In this approach, the free energy is expanded in the interaction strength. Thus, the zero order term is the free energy of two ideal Bose gases.

To formulate this quantity, we define the average thermal de Broglie wavelength

 \lambdabar = (λ0λ2J)1/3, (8)

with and . The density of excited atoms for each of the minima is

 ne,j=1\lambdabar3G3/2(zj), (9)

with , and being the standard Bose function. is the fugacity and is computed by the inverse function of Eq. (9) in the free energy below. Here we formally introduce two chemical potentials , but for the result further down we preserve only the total density. The total density of this system is , where is the density for the th component of the system, and is fixed in free energy computation below. The density of the ground states is defined as , which leads to a condensate density of , if . Therefore, the free energy is given by with

 A0,iV=⎧⎪ ⎪⎨⎪ ⎪⎩−kBT\lambdabar3G5/2(zi)+nikBTln(zi)if zi<1,−kBT\lambdabar3G5/2(1)if zi=1, (10)

and being the volume of the system.

We now compute the next order of the free energy which is due to and linear in the interaction term (II). We approximate and , according to the HYL approximation. Therefore, the first order contribution to the free energy is

 AIV=V02[2n21+2n22−n20,1−n20,2]+V12n1n2. (11)

We explore the properties of this approximation of the free energy for the experimental parameters of Ref. Struck13 (), for various ratios of and temperatures, and with fixed total density . In units of the lattice constant nm and the effective tunneling parameter nK, the interaction strength is given by , and we have a density of of Rb atoms.

At low temperatures, the density of the two condensates becomes non-zero, which implies a U symmetry breaking. To analyze the critical behavior of the chiral phase transition, we consider the magnitude of the free energy relative to the -symmetric state with

If becomes negative as the temperature is lowered, it indicates that the symmetry is broken. Furthermore, where the new minima emerge indicates if the phase transition is first or second order.

We find that for the minimum of is always located at , indicating that the symmetry is preserved. However, for , we find that the symmetry breaks, as new minima appear away from the symmetric point. The temperature at which these minima occur is the critical temperature of the symmetry breaking. It is plotted against the ratio in Fig. 2 (a). We also find that the order of the symmetry breaking changes as is varied. As shown in Fig. 2 (b), the two minima of emerge continuously at , indicating a second order phase transition in the regime of . In this regime, as shown in Fig. 2 (a), the critical temperature of the symmetry breaking only increases weakly with increasing . However, for , the minima of emerge away from the symmetric point at , as demonstrated in Fig. 2 (c), indicating a discontinuous phase transition. The critical temperature rapidly increases with increasing , as illustrated in Fig. 2 (a). For , develops a plateau at , below the critical temperature, as shown in Fig. 2 (d). This reflects the emergent symmetry at this interaction strength, which corresponds to a Heisenberg fixed point, as we discuss below.

We again emphasize that this analysis suggests a first transition of the system, because the bare value of the interaction is . Furthermore, as we discuss in the next section, our RG analysis suggests that the first order regime extends throughout the entire regime .

We note that the and the symmetry breaking occur at the same temperature, within the HYL approximation. This occurs because the condensate density is responsible for generating the two minima in the free energy. We demonstrate this computing the condensate fraction versus temperature in Fig. 3 (a) for and (b) respectively. The critical temperature of the symmetry breaking is the same as the temperature at which becomes non-zero. The first- and second-order phase transitions of the simultaneous and symmetry breaking are indicated by the discontinuous and continuous change of the condensate density in Fig 3.

## Iv Renormalization Group approach

### iv.1 Renormalization Group flow

To study the phase diagram systematically, we perform a renormalization group calculation at one-loop order in the weak-coupling regimeStoof09 (). We write the partition function as a path integral:

 Z=∫D[ϕ∗,ϕ]e−S[ϕ∗,ϕ]/ℏ, (13)

with the action

 (14)

and in the coherent-state representation of the Hamiltonian . We consider the classical limit of the effective action , i.e. we ignore the dependence of the fields on the Matsubara time , . This is equivalent to only taking the Matsubara frequency into account Stoof09 (). In this approximation, the partition function simplifies to

 Z=∫D[ϕ∗,ϕ]e−βFL[ϕ∗,ϕ], (15)

where is the Landau free energy functional with

 FL,0=∫d3R∑j=1,2[ℏ22m∗∣∣∇ϕj∣∣2−μ∣∣ϕj∣∣2], (16)

and the interaction

 (17)

We rescale the momenta in the - and -direction by to arrive at a spatially homogeneous expression, with the effective mass and . To perform the RG transformation, we introduce an energy cutoff of this system with the momentum cutoff , which sets the maximum energy scale of the field theory description. A physical choice for the energy cut-off is the bandwidth of the dispersion in the x-y plane, which is determined by .

We note that in the following we assume that the low-energy regime of the driven system is approximately given by . The assumption is based on the high-frequency lattice shaking in our system, where the lattice shaking frequency is and the bare hopping amplitude is i.e.  Struck13 (). Though from the eigenstate thermalization hypothesis all the Floquet eigenstates are indistinguishable from the infinite-temperature state or a completely random state at long-time limit Breuer (); Russomanno (); Hone (); Iadecola (); Rigol (); Moessner (); Shirai (); Liu (); Abanin (), in this study we are interested in the quasi-stationary(or metastable) state during high-frequency driving. Recent theoretical study shows for an isolated Floquet systems in high-frequency driving with local interactions, a nonintegrated system as we considered here, heating will be exponentially slow. It implies that prethermalized Floquet many-body phases, though metastable, will be very long-lived Abanin2 (). On the other hand, because the energy is almost conserved up to timescale , the system will first reaches a quasi-stationary state with a finite temperature Saito (); Kuwahara (). Thus, thermal equilibrium due to the approximative character of the Floquet Hamiltonian is well-approximated for the large frequencies and the experimentally relevant times considered. Such that, the ensemble average of the periodically driven isolated systems in the intermediate timescale is approximated by the ensemble average of the quasi-stationary state(or Floquet Hamiltonian) in this study. The above concept is also applied in recent theoretical studies Demler (); Bukov ().

In an RG step, we split the fields into a low-energy and a high-energy part. The high-energy degrees of freedom are integrated out to obtain a renormalized free energy functional with renormalized chemical potential and coupling constants and , which we compute at one-loop order. Iterating this RG step results in a set of flow equations, given below. The derivation of the flow equations using the -expansion is discussed in Appendix A.

We define the dimensionless variables , , , . We note that by construction all of these dimensionless quantities are small compared to , because they are divided by the high energy cut-off . For this reason, we terminate the flow if one of these parameters reaches unity, as described below. In terms of these variables, the flow equations to linear order in are

 dμΛdl =2μΛ−2g0TΛ(1+μΛ)−g12TΛ(1+μΛ), (18) dg0dl =εg0−5g20TΛ−g212TΛ, (19) dg12dl =εg12−2g212TΛ−4g0g12TΛ. (20)

where is the logarithm of the ratio between the initial momentum cutoff and the running cutoff . We note that by setting , these flow equations reduce to the two standard RG equations of the one-component, complex theory.

To identify the fixed points of the flow, we set the Eqs. (18), (19) and (20) to zero. This results in three fixed points, which are given by

 μ∗Λ=0,g∗0=0,g∗12=0, (21) μ∗Λ=ε5,g∗0=ε5TΛ,g∗12=0, (22) μ∗Λ=ε4,g∗0=ε6TΛ,g∗12=ε6TΛ, (23)

to linear order in . For these points, which are illustrated in Fig. 4, the variables remain unchanged under the RG flow. We investigate the flow further by expanding the flow equations to first order around the fixed points (see Appendix B).

This expansion shows that the non-interacting fixed point (21) is unstable in all directions. In the vicinity of the second fixed point (22), we recover the flow in the plane Stoof09 (), which includes one relevant and one irrelevant direction. In addition we find one more relevant direction, which drives the system away from the plane. In Fig. 4, we qualitatively sketch the flow diagram in the vicinity of three fixed points. The irrelevant and relevant directions obtained in the fixed-point analysis correspond to the flow toward the fixed points and away from it, respectively. On the plane, the line through and corresponds to the boundary between the thermal phase and the condensate phase. Below and above this line, the chemical potential flows to large negative and positive values, corresponding to symmetry preserving and breaking, respectively. For non-zero values of , we find that increases under the RG transformation. This indicates that the inter-component interaction is always relevant in a two-component Bose gas.

The flow behavior in the vicinity of the third fixed point (23) is also depicted in Fig. 4. We refer to this fixed point as a Heisenberg fixed point, because it displays symmetry. The fixed-point analysis shows that there is one irrelevant direction along the plane, which is similar to the behavior close to the second fixed point on the plane. In addition, there is a relevant direction, that drives the chemical potential to . The line through and on the plane is the boundary of the transition from the thermal to the condensed phase. This line is part of the critical surface, illustrated as the transparent green plane in Fig. 4, below which the chemical potential flows to , and above which it flows to . We note that this surface bends down to negative values of and , which is a prerequisite for the first order transition for , discussed below.

Finally, the flow behavior perpendicular to the plane shows marginal behavior in the first-order analysis near the third fixed point, see Appendix B. We integrate the RG equations to study the RG flow further. In Fig. 5, we depict the RG flow in the - plane, that is created by the Eqs. (18), (19) and (20). As an example, we fixed the initial value of the chemical potential to . We choose several initial values of to depict the trajectories, to illustrate the flow.

We also indicate the three fixed points, as well as whether the chemical potential flows to , or towards , via color coding: The orange colored region corresponds to the regime, whereas the green and purple regions indicate . The latter regime is further divided into two subregimes, which are distinguished by the asymptotic behavior of and under the flow. If the flow approaches the fixed point asymptotically, the initial point is colored green. This occurs for . If the flow is repelled by this fixed point, and asymptotically flows to a negative value, the initial point is colored purple. This occurs for .

For the regime , the critical behavior is controlled by the fixed point at . The flow towards to this point separates into an initial, fast flow onto the marginal surface, connecting the second and third fixed point, on which it flows logarithmically slow, when parametrized by . The chemical potential on the other hand diverges exponentially fast to either . When its absolute value reaches unity, the flow moves out of its range of validity, and we terminate it. Therefore, the RG flow, while asymptotically moving towards the Heisenberg fixed point, predicts a physical state for which . This suggests that the system breaks both symmetries, while the is preserved. The perturbation is therefore dangerously irrelevant: It modifies the ground state of the system in a qualitative way from the state that it would have if the magnitude of was identically zero. The second order phase boundary between this phase with two condensates, each breaking a symmetry, and the thermal phase, is indicated by the green and orange circles. This phase boundary shifts to larger values of and , when the initial value of is increased. When it is reduced, the boundary shifts to smaller values. This indicates that the condensation transition can tolerate larger repulsive interactions, when the chemical potential is larger.

For , the parameters are initially attracted to a line that is approximately perpendicular to the line , as visible in Fig. 5. Once in the vicinity of this line, the parameters are now repelled by the Heisenberg fixed point. In particular, is renormalized to a negative value, while increases and remains positive. This indicates the breakdown of the quartic, effective field theory, as the energy of the system is no longer bounded from below. Higher order terms, such as a term:

 F6 ∼ g6∫dR(|ϕ1|6+|ϕ2|6) (24)

have to be included to provide a stable description of the system. Furthermore, it indicates that a first-order phase transition can occur, Ref. Domany (). We note that this term is an effective three-body interaction. It is generated by integrating out high-energy excitations, and results from virtual two-body collisions. For a deep optical lattice, this has been discussed in Refs. Johnson (); Will (). This scenario is demonstrated in Fig. 6. For a first order transition to occur in this system, an effective free energy with three distinct minima has to emerge, similar to the example in Fig. 2 (c ). For this to occur, the chemical potential has to flow to a negative value, as demonstrated above, has to flow to a negative value, as shown above, and finally the magnitude of , and has to be such that three minima occur, and the side minima emerge as the global minima. As can be checked, for this is required.

In Fig. 6 we show the - plane, where we set , which is the plane relevant to the triangular lattice system. We depict the magnitude of for , for which . Near the critical surface, is negative for any initial value. Furthermore, we depict the regime for which . This, and any other choice for , results in a narrow regime below the entire critical surface. It is this scenario that suggests a first order transition. We note that a bosonic system of constant density has a monotonously increasing chemical potential, as the temperature is lowered. When the chemical potential reaches the critical surface, the system condenses. For the system of two bosonic fields, and for , however, before the chemical reaches the critical surface, the effective action develops side minima, which results in a first order transition.

### iv.2 Phase diagram

In Figs. 7 and 8 we show the resulting phase diagram of the system. In Fig. 7 we keep the chemical potential and fixed, and vary and the temperature. For high temperatures, the system is in the thermal gas phase, of any value of . This regime is labelled as orange circles. As the temperature is lowered, the system condenses, either into a phase of two condensates, each breaking a symmetry, or into a chiral condensate phase, which breaks a and the symmetry. This transition occurs at , as illustrated as the red dashed line. Furthermore, the order of the phase transition changes, as is increased. For the system undergoes a second order phase transition, for a first order transition. We also note that the critical temperature is reduced with increasing . To illustrate the influence of the inter-component interactions on the critical temperature further, we show the phase diagram in the - plane in Fig. 8. We note that the phase diagram for the interaction found in the experiment Struck13 () is illustrated versus temperature and chemical potential in Fig. 8(c).

## V Conclusions

In this article, we have studied the critical behavior of Bose-Einstein condensation in a frustrated, triangular lattice. Using a renormalization group approach, we have demonstrated that the phase transition is of first order. At this transition, the system breaks both a symmetry and a , simultaneously, resulting in a condensation with chiral order. We achieve this insight by mapping the system onto a complex theory consisting of two complex fields. This field theory has a symmetry, corresponding to the symmetry of each complex field, as well as a symmetry of exchanging the two fields. We demonstrate that the critical behavior of this effective field theory is controlled by the ratio of the inter-component and the intra-component interaction strength. For the system undergoes a second order phase transition, in which both symmetries are broken, while preserving the symmetry. For the system undergoes a first transition, in which one symmetry and one symmetry are broken. The latter scenario is reflected in the renormalization group as follows: Rather than being attracted by a fixed point of the flow, which controls the critical behavior, the parameters of the system are repelled by a fixed point, and the emergent free energy becomes unbounded from below. By considering the properties of this flow, this instability can be identified with a first order transition, because the parameters are renormalized in such a way that additional side minima can appear, as in a theory, and eventually become the global minima. Experimentally, the first order character of the transition could be observed by measuring the condensate fraction and the chiral magnetization as a function of temperature. This intriguing critical behavior, in which condensation occurs as a first order transition, could also occur in numerous other cold atom systems, in which the dispersion of the system has more than one minimum. Our study would apply in an analogous manner, and would therefore be of crucial guidance to understand the critical behavior in such systems. In the end, we note that our analysis might be motivated primarily by the Sengstock experiment, but that this type of analysis similarly applies to other frustraed systems Wirth11 ().

###### Acknowledgements.
We acknowledge support from the Deutsche Forschungsgemeinschaft through the SFB 925 and the Hamburg Centre for Ultrafast Imaging, and from the Landesexzellenzinitiative Hamburg, which is supported by the Joachim Herz Stiftung. WMH acknowledges support from the Ministry of Science and Technology in Taiwan through grant MOST104-2112-M-005-006-MY3.

## Appendix A Derivation of the flow equations

In this Appendix, we derive the renormalization group equations for the free energy functional of interest (16,17). For reasons of clarity, we rewrite the free energy according to:

 FL,0 =∫d3R∑j=1,2[ℏ22m∗∣∣∇ϕj∣∣2−μj∣∣ϕj∣∣2], (25) FL,I =∫d3R(V0,12|ϕ1|4+V0,22|ϕ2|4+V12|ϕ1|2|ϕ2|2). (26)

This will make it easier to identify distinct terms in the perturbative expansion later on. At the end of the calculation, we will set and . Pictorially, we can represent the two order parameters and as colored lines, where the propagator corresponds to the respective chemical potentials and the coupling constants appear as the elementary vertices of the theory (Fig. 9).

Assuming the interactions to be weak, the corrections to the propagators and vertices can be obtained using the cumulant expansion of the renormalized free energy functional Stoof09 (); Wilson ().

The first-order correction is given by

 ⟨FL,I⟩0,>, (27)

where the expectation value is defined as

 ⟨⋯⟩0,>≡1Z1,>Z2,>× ∫∫d[ϕ∗1,>]d[ϕ1,>]d[ϕ∗2,>]d[ϕ2,>](⋯)× e−β∫d3R(ℏ22m1∣∣∇ϕ1,>∣∣2−μ1∣∣ϕ1,>∣∣2)× e−β∫d3R(ℏ22m2∣∣∇ϕ2,>∣∣2−μ2∣∣ϕ2,>∣∣2), (28)

with normalization constants and .

To compute the first-order corrections, we split the two complex order parameters into a high-energy and a low-energy part

 ϕi=ϕi,<+ϕi,>with i=1,2, (29)

where:

 ϕi,<=∑k<Λ/bϕi,kei%kR√Vwith i=1,2, (30) ϕi,>=∑Λ/b

Here, is the volume of the system, is the high-energy cutoff of the theory and is some number bigger than one, such that the new cutoff after one RG step is lowered compared to the old one. is also referred to as the running cutoff. As can be seen from (28), the Fourier components of the order parameters that correspond to high-energy excitations are integrated out to obtain corrections for the renormalized free energy functional.

For each of the three addends in (27), the splitting of the order parameter results in terms. Most terms vanish because of the Gaussian integration in (28), since they involve an odd number of high-energy parts of the order parameter. Also, must always appear in combination with its complex conjugate to yield a non-zero contribution. Of the remaining terms, only those are of interest that contribute corrections to the chemical potentials (in contrast to just shifting the renormalized free energy functional by a constant factor). Therefore, the only relevant terms for the flow equations are:

 4V0,12 ∫d3Rϕ1,<ϕ∗1,<⟨ϕ∗1,>ϕ1,>⟩0,>, (32) 4V0,22 ∫d3Rϕ2,<ϕ∗2,<⟨ϕ∗2,>ϕ2,>⟩0,>, (33) V12 ∫d3Rϕ2,<ϕ∗2,<⟨ϕ∗1,>ϕ1,>⟩0,>, (34) V12 ∫d3Rϕ1,<ϕ∗1,<⟨ϕ∗2,>ϕ2,>⟩0,>. (35)

We can see that the -interaction mixes in corrections from to and vice versa. The remaining evaluation of the expectation value can be carried out in Fourier space straightforwardly

 ⟨ϕ∗i,>ϕi,>⟩0,> =1V∑Λ/b

where, in the last step, the continuum limit of the Fourier sum is taken in dimensions and . To mimic our original system, a rescaling of the Fourier components must be performed after one RG step. The shell in Fourier space that is integrated out is taken to be infinitesimally thin with thickness and area , where and is the Gamma function. With this, the change in and after one RG step caused by the corrections (32-35) can be expressed as

 dμi= −2V0,iΛd(2π)d2πd/2Γ(d2)kBTe−2lεi,Λ−μie−lddl −V12Λd(2π)d2πd/2Γ(d2)kBTe−2lε¯i,Λ−μ¯ie−lddl, (37)

where, as before, is either one or two and is the negation of . These first-order corrections are depicted in Fig. 10.

To obtain the desired flow equations, we have to rescale the chemical potentials and the coupling constants according to:

 μi →μie−2l, V0,i →V0,ie−(4−d)l, V12 →V12e−(4−d)l. (38)

The flow equations for and take the form:

 dμidl=2μi −2V0,iΛd(2