TUMHEP711/09
MPP200917
Electroweak and Flavour Structure of a
Warped Extra Dimension with Custodial Protection
Michaela E. Albrecht, Monika Blanke, Andrzej J. Buras,
Björn Duling and Katrin Gemmler
Physik Department, Technische Universität München, D85748 Garching, Germany
MaxPlanckInstitut für Physik (WernerHeisenbergInstitut),
D80805 München, Germany
TUM Institute for Advanced Study, Technische Universität München,
D80333 München, Germany
We present the electroweak and flavour structure of a model with a warped extra dimension and the bulk gauge group . The presence of implies an unbroken custodial symmetry in the Higgs system allowing to eliminate large contributions to the parameter, whereas the symmetry and the enlarged fermion representations provide a custodial symmetry for flavour diagonal and flavour changing couplings of the SM boson to lefthanded downtype quarks. We diagonalise analytically the mass matrices of charged and neutral gauge bosons including the first KK modes. We present the mass matrices for quarks including heavy KK modes and discuss the neutral and charged currents involving light and heavy fields. We give the corresponding complete set of Feynman rules in the unitary gauge.
1 Introduction
Models with a warped extra dimension, also called RandallSundrum (RS) models [1, 2, 3, 4], in which all Standard Model (SM) fields are allowed to propagate in the bulk, offer natural solutions to many outstanding puzzles of contemporary particle physics. In addition to providing a geometrical solution to the hierarchy problem related to the vast difference between the Planck scale and the electroweak (EW) scale, they also allow to naturally generate hierarchies in fermion masses and weak mixing angles [5, 6], suppress flavour changing neutral current (FCNC) interactions [7, 8, 9], construct realistic models of EW symmetry breaking (EWSB) [10, 11, 12, 13, 14, 15] and achieve gauge coupling unification[16, 17].
The question then arises whether some imprints of this new physics scenario could be in the reach of the LHC, while satisfying all existing experimental constraints, coming in particular from EW precision tests, from the data on FCNC processes in both quark and lepton sectors and also the data on the very highly suppressed electric dipole moments (for recent reviews, see [18, 19, 20]).
A necessary, though not always sufficient, condition for direct signals of RS models at the LHC is the existence of KaluzaKlein (KK) modes with masses. Early studies of EW precision observables (EWPO) [10, 21] have shown that with the SM gauge group in the bulk such low masses of KK particles are inconsistent in particular with the bounds on the oblique parameter and the wellmeasured coupling.
In a number of very interesting papers [10, 11, 12, 22, 14, 15, 13] these two obstacles have been basically overcome by enlarging the bulk symmetry to
(1.1) 
and enlarging the fermion representations, so that the discrete leftright symmetry , exchanging and , is preserved. The presence of the additional gauge group implies the existence of an unbroken custodial symmetry in the Higgs sector, so that tree level contributions to the parameter can be safely neglected. The symmetry and the related enlarged fermion representations eliminate the problematic contributions to the coupling.
Interestingly, the presence of new light KK modes necessary to solve the “ problem” implies significant contributions to the parameter at the one loop level [23]. However with an appropriate choice of quark bulk mass parameters, an agreement with the EW precision data in the presence of light KK modes can be obtained [24, 25]. In fact, while the masses of the KK gauge bosons are forced to be at least to be consistent with the data on the oblique parameter , fermionic KK modes with masses even below can be made consistent with the measured EWPO.
The suppression of FCNC transitions to an acceptable level in the presence of light KK modes turns out to be much more challenging if the hierarchy of fermion masses and weak mixings is supposed to come solely from geometry so that the fundamental 5D Yukawa couplings are anarchic. In fact, recent studies demonstrate that in this case the data on the CPviolating parameter imply a lower bound on the lightest gauge KK modes in the ballpark of [26, 27], the corresponding bound from is above [28, 29]^{1}^{1}1We would like to mention that this bound can be avoided by new choices of lepton representations [30]. and even stronger bounds come from electric dipole moments [8, 31]. Moreover it has been pointed out in [32] that the flavour problem in these models becomes even more serious when and decays are considered simultaneously. Note however that the bound in question can be somewhat relaxed by appropriately chosen brane kinetic terms [26] and/or by allowing the Higgs boson to propagate in the bulk [32].
In view of this situation a number of proposals has been made in order to overcome these “FCNC problems” of RS models that are directly related to the breakdown of the universality of gauge boson–fermion couplings implied by the geometric explanation of the hierarchical structure of fermion masses and mixings. This breakdown implies the violation of the GIM mechanism [33] and consequently tree level FCNC transitions that are inconsistent with the data for light KK scales, provided that anarchic 5D Yukawa couplings are chosen and the relevant couplings are .
In [34] a class of RS models has been considered that makes use of bulk and brane flavour symmetries in order to prevent the theory from large FCNCs. It has been shown that if flavour mixing is introduced via UV brane kinetic terms, the GIM mechanism is realized and a minimal flavour violating (MFV) model [35, 36, 37, 38, 39] can be obtained. However, the natural explanation of fermionic hierarchies had to be abandoned in that setup. A different strategy has been followed in [40], where the field theoretical concept of MFV has been promoted to the 5D theory, i. e. the bulk mass matrices are expressed in terms of the 5D Yukawa couplings. Low energy flavour violation can be further suppressed by a single parameter that dials the amount of violation in the up or down sector. If this parameter is ensured to be small, no flavour or CP problem arises even with KK masses as low as . A more thorough analysis, including the presentation of a possible dynamical origin of such a model, has been given in [41]. Another economical model based on a bulk flavour symmetry has been proposed in [42]. Here the righthanded down quark bulk masses are enforced to be degenerate, so that the contributions of the operator to are generated only by suppressed mass insertions on the IR brane. A recent approach [43] presents a simple model where the key ingredient are two horizontal symmetries. The SM fields are embedded into the 5D fields motivated by protecting . The horizontal symmetries force an alignment of bulk masses and down Yukawas which strongly suppresses FCNCs in the down sector. FCNCs in the up sector, however, can be close to the experimental limits.
In two recent papers [27, 44] we took a different strategy and investigated to which extent a hierarchy in the 5D Yukawa couplings has to be reintroduced in order to achieve consistency with the existing data on FCNC processes in the presence of KK modes in the reach of the LHC. In particular in [27] we have demonstrated that there exist regions in parameter space with only modest finetuning in the 5D Yukawa couplings involved which allow to obtain a satisfactory description of the quark masses and weak mixing angles and to satisfy all existing and EW precision constraints for scales in the reach of the LHC. As the dominant part of the observed hierarchy in masses and mixings is still explained through the AdS geometry, the resulting hierarchies are significantly milder than in the SM and other usual 4D approaches.
Subsequently, confining the numerical analysis to the regions of parameter space allowed by observables and with only modest finetuning, we have presented in [44] a complete study of rare and meson decays including , , , , , , and .
In this context it should be emphasised that the presence of FCNC transitions already at the tree level in the model in question, as opposed to the MSSM and Little Higgs models, necessarily implies other patterns in CPviolating observables and rare decay branching ratios. In particular in RS models not only nonMFV interactions are present, like for instance in the Little Higgs models with TParity, but also new operators become important that are strongly suppressed in the latter. As found in [27, 44] such new contributions lead to interesting deviations from the SM and in particular from models with Constrained MFV [35, 36, 45] in observables that are still poorly measured and which allow for large new physics contributions.
The main results of [27] can be briefly summarised as follows:

The EW tree level contributions to observables mediated by the new weak gauge boson , while subleading in the case of and , turn out to be of roughly the same size as the KK gluon contributions in the case of physics observables.

The symmetry implies automatically the protection of flavour violating couplings so that tree level contributions to all processes in which flavour changes appear in the down quark sector are dominantly represented by couplings.

However, the tree level contributions to processes are of higher order in and can be neglected.
On the other hand the main messages from [44] are as follows:

New physics contributions to rare and decays, as opposed to transitions, are governed by tree level contributions from boson exchanges (dominated by couplings) with the new heavy EW gauge bosons playing a subdominant role.

Imposing all existing constraints from transitions we find that a number of branching ratios for rare decays can differ significantly from the SM predictions, while the corresponding effects in rare decays are modest. In particular the branching ratios for and can be by a factor of three and two larger than the SM predictions, respectively. The latter enhancement could be welcomed one day if the central experimental value [53] will remain in the ballpark of and its error will decrease.

However, it is very unlikely to get simultaneously large NP effects in rare decays and , which constitutes a good test of the model.

Sizable departures from the MFV relations between and and between and the decay rates are possible.

The pattern of deviations from the SM differs from the deviations found in the LHT model [54].
It is interesting that in spite of many new flavour parameters present in this model a clear pattern of new flavour violating effects has been identified in [27, 44]: large effects in transitions, large effects in rare decays, small effects in rare decays and the absence of simultaneous large effects in the and system. This pattern implies that an observation of a large asymmetry would in the context of this model preclude sizable NP effects in rare decays. On the other hand, finding to be SMlike will open the road to large NP effects in rare decays, even if such large effects are only a possibility and are not guaranteed. On the other hand, an observation of large NP effects in rare decays would put this model in serious difficulties.
In [27, 44] only a brief description of the RS model in question has been presented as only gauge boson exchanges were relevant at the tree level. In particular details on the fermion sector have not been presented there. For the subsequent phenomenological studies like the and transitions it is of interest to have a more detailed presentation which is the main goal of our paper. We formulate a particular RS model based on the bulk gauge group in (1.1) and having appropriate quark representations in order to avoid tensions with EWPO. We work out the general structure of the gauge and fermion sectors, discuss the new sources of flavour violation, and we give a collection of Feynman rules^{2}^{2}2Some of these Feynman rules have already been presented in [55, 56]. that can be used to calculate all observables of interest. In fact a subset of the Feynman rules presented here has already been used in [27, 44].
Throughout our analysis we follow the perturbative approach, i. e. we first solve the 5D equations of motion and perform the KK decomposition in the absence of EWSB, as also done e. g. in [55, 57] and then treat the Higgs vacuum expectation value (VEV) as a small perturbation that induces mixing among the various modes. The complementary approach, solving the equations of motion already in the presence of EWSB, has been followed e. g. in [58, 59, 60, 61]. Recently, a very detailed theoretical discussion of the latter approach has been presented in [62]. In Appendix C.2 we show that both approaches are indeed equivalent; for an independent discussion see also [63].
The present paper is organised as follows. In Section 2 we present in detail the gauge sector of the model and in particular the effects of EWSB. The final formulae for gauge boson masses and mixings in the charged and neutral sectors are collected in Appendix C. Next in Section 3 we set up the quark representations under the bulk gauge group. In Section 4, one of the main sections of our paper, we work out the flavour structure of the quark sector. After a detailed discussion of quark mass matrices and Yukawa couplings in the flavour eigenbasis we outline the diagonalisation of these matrices and study the structure of weak neutral and charged currents. Subsequently the couplings of KK gluons and photons are considered. This section forms the basis of the Feynman rules in the quark sector that are collected in Appendix D. We end this section by listing the sources of flavour violation in this model, with the pattern of flavour violation, in particular in processes, governed by the custodial protection present in the model. In Section 5 we list the parameters of the model and present a useful parameterisation for the 5D Yukawa couplings in terms of parameters accessible at low energies. In Section 6 we discuss one possible realisation of the lepton sector and present a dictionary that allows in a straightforward manner to obtain the Feynman rules for the leptons from those of quarks. We close the paper with a brief summary in Section 7.
2 Gauge Sector
2.1 Preliminaries
We consider an gauge theory on a slice of AdS with the metric [1]
(2.1) 
with the fifth coordinate being restricted to the interval , and . In order to simplify the phenomenological discussion in [27, 44] we chose to work with the sign convention for the metric, i. e. . The gauge bosons and fermions are allowed to propagate in the 5D bulk, while the Higgs field will be localised on or near the IR brane ().
In the EW sector, we consider the gauge symmetry [10, 11, 22]
(2.2) 
where is the discrete symmetry interchanging the two groups. This means for instance that . The gauge group (2.2) is broken by boundary conditions (BCs) on the UV brane () to the Standard Model (SM) gauge group, i. e.
(2.3) 
This breakdown is achieved by the following assignment of BCs^{3}^{3}3These BCs can be naturally achieved by adding a scalar doublet with charge on the UV brane, that develops a VEV (see [61, 3] for details).
(2.4)  
(2.5) 
where the first (second) sign denotes the BC on the UV (IR) brane: stands for a Neumann BC while stands for a Dirichlet BC. Furthermore and . The fields and are given in terms of the original fields and as follows:
(2.6)  
(2.7) 
where
(2.8) 
Here, and are the 5D gauge couplings of and , respectively. Note that the BCs for a gauge field imply automatically opposite BCs for its 5th component . In what follows we choose to work in the gauge and .
The fields with BCs have, in addition to the massive KK modes, zero modes which are massless at this stage and are identified with the SM gauge bosons and of . The fields with BCs contain only massive KK modes.
Before EWSB the profiles of gauge boson zero modes along the extra dimension are flat. The profiles of KK gauge bosons are given by [6] (see also Appendix B for details)
(2.9) 
where and are the Bessel functions of first and second kind, and explicit expressions for and can be found in Appendix B. The bulk masses are approximately given by [6]
(2.10) 
for the modes with a BC on the IR brane that we are presently interested in. The accuracy of this approximate formula improves significantly with increasing , hence for the first KK modes it is safer to work with the exact KK masses. These can be found numerically to be
(2.11) 
for gauge bosons with BCs, and
(2.12) 
for gauge bosons with BCs. Here we have introduced the effective new physics scale and set in order to solve the hierarchy problem. The suppression in the latter case is a direct consequence of the different BC on the UV brane [55]. Note that the KK masses for the gauge bosons depend neither on the gauge group nor on the size of the gauge coupling, but are universal for all gauge bosons with the same BCs. Only after EWSB, the weak KK gauge boson masses will receive small additional corrections. As can easily be seen from (2.9), the gauge KK modes are localised near the IR brane.
To further proceed it will be useful to follow [55] and define the fields
(2.13) 
and
(2.14)  
(2.15) 
where again is given in terms of gauge couplings (see (2.8) for the definition of )
(2.16) 
Because of the mixing between the various gauge boson zero and KK modes , but corrections appear first at order . Their impact on EW precision studies is beyond the scope of this paper and will be studied elsewhere.
We note that the above relations can be modified by the presence of additional gauge kinetic terms on the UV and IR branes, that are allowed by the symmetries of the model. In order not to complicate our analysis, we will neglect such terms and work exclusively with the action given in Appendix A. A generalisation of our results to include also the effects of possible brane terms is straightforward. In Section 4.5.7 we comment on the effects of such terms on flavour phenomenology.
2.2 Electroweak Symmetry Breaking
As discussed in the previous section, the bulk gauge symmetry in (1.1) is broken to the SM gauge group
(2.17) 
by means of the BCs of the EW gauge bosons on the UV brane. In order to achieve the standard EWSB, , a Higgs boson is introduced that is localised either on or near the IR brane, transforming as a selfdual bidoublet of
(2.18) 
and being a singlet under , . In the case of a 5D Higgs field living in the bulk, the whole bidoublet has to obey BCs in order to yield a light zero mode.
When its neutral component develops a 4D effective VEV, on or near the IR brane the symmetry breaking
(2.19) 
takes place. We see explicitly that in the Higgs sector of the theory an unbroken custodial symmetry remains, being responsible for the protection of the parameter. Similarly the symmetry, protecting the coupling, remains unbroken.
Combining then the symmetry breakings by BCs on the UV brane and by the Higgs VEV in the IR, we see that the low energy effective theory is described by the spontaneous breaking
(2.20) 
as required by phenomenology. The symmetry breaking structure of the model is displayed in Fig. 1.
Now due to the unbroken gauge invariance of QED and QCD, the gluon and photon fields including their KK modes do not couple to the Higgs boson at leading order in perturbation theory and hence do not mix with each other or with , and and the higher KK modes of and . Therefore, even after EWSB
(2.21)  
(2.22) 
and the corresponding states remain mass eigenstates. On the other hand the kinetic term for the Higgs field (see Appendix A)
(2.23) 
leads to corrections to the masses of , and as well as of , and , and mixing between states of the same electric charge is induced. Here
(2.24) 
where is the Higgs shape function along the extra dimension. We assume to be of the form
(2.25) 
where in the limit the case of an IR brane localised Higgs is recovered. The case of a bulk Higgs has first been considered in [64, 65]. Furthermore
(2.26) 
and denotes the effective 4D VEV of the zero mode of in (2.18).
Restricting the discussion to for simplicity, the gauge boson interactions with the Higgs resulting from (2.23) lead to two mass matrices and [55]
(2.27)  
(2.28) 
with and given explicitly in Appendix C.
In order to determine the physical mass eigenstates and the corresponding masses, and have to be diagonalised by means of orthogonal transformations:
(2.29)  
(2.30) 
The mass eigenstates and are then related to the gauge eigenstates of the KK modes via
(2.31) 
The explicit form of the orthogonal matrices and can be found in Appendix C.
3 Fermion Sector – Quarks
3.1 Preliminaries
In order to preserve the symmetry, that is necessary for the suppression of dangerous contributions to EW precision observables [10, 11, 22, 14, 15, 23], we will choose a particular simple set of representations of the group. Although in order to satisfy EW precision measurements only the third quark generation needs to preserve the symmetry, the incorporation of CKM mixing requires the same choice of representations also for the first two quark generations. This is crucial for having a custodial protection for the flavour violating couplings [44] as well.
In this section we restrict our attention to the quark sector of the model. The lepton sector will be discussed separately in Section 6.
The particular fermion assignment given below has been motivated by the analyses of [13, 15, 14, 23, 66]. In particular the representations given below can easily be embedded into complete multiplets used in [14, 23, 66] in the context of models with gaugeHiggs unification.
We introduce three multiplets per generation :
(3.1)  
(3.2)  
(3.3) 
The corresponding states of opposite chirality are given by
(3.4)  
(3.5)  
(3.6) 
The following comments are in order:

and are bidoublets of , with acting vertically and horizontally.

and are singlets of .

transform as under . The embedding of the righthanded downtype quarks into triplet representations is necessary in order to allow for a invariant Yukawa coupling.

The charges assigned to the various multiplets are charges.

The charges assigned to separate fields are electric charges, given as
(3.7) where and denote the third component of the and isospins, respectively.

Only the fields obeying BCs have massless zero modes. Up to small mixing effects with other massive modes due to the transformation to mass eigenstates discussed in Section 4, these zero modes can be identified with the usual SM quarks.

The remaining fields are KK modes with approximately vectorlike couplings. We thus have in this model additional heavy fermionic states. These are
(3.8) (3.9) (3.10) where .

Left and righthanded fermion fields are defined via .
3.2 KK Decomposition and Bulk Profiles
3.2.1 Zero Modes
The profiles of lefthanded fermionic zero modes with respect to the flat metric are given by [5, 6]
(3.11) 
where is the bulk mass of the 5D fermion field. In the case of righthanded zero modes, has to be replaced by in the above formula and in the discussion following below. We note that

For the normalisation factor in (3.11) is and is peaked around , i. e. fermions with bulk mass parameter are placed close to the UV brane.

For , as is the case of the top quark, the second term in the denominator of (3.11) can be neglected and we obtain
(3.12) Thus the shape function is strongly peaked towards , i. e. the IR brane.

One should stress that generally the and of left and righthanded SM fermions can differ from each other, as these fermions are zero modes of different 5D representations. As both and enter the formula for the Yukawa couplings and fermion masses, this freedom can help to satisfy certain features in EW precision studies [13, 15, 14, 23] and in flavour physics [8, 62] while keeping the fermion masses of their natural size. This is in particular relevant for the third quark generation.
3.2.2 KK Modes
The shape functions for fermionic KK modes with respect to the warped metric are given by [6] (see also Appendix B for details)
(3.13) 
where for left(right)handed modes, and expressions for and can be found in Appendix B. The KK masses are approximately given by
(3.14) 
where the sign corresponds to a BC for the lefthanded fermion on the IR brane. Again, as in the case of gauge KK modes, the accuracy of (3.14) improves with increasing .
The following comments are in order [6, 4]:

The bulk mass parameter is universal for the full tower of KK modes and equal to the describing the localisation of the zero mode if such mode exists.

In spite of the same even for the form of (3.13) implies that all KK modes are localised near the IR brane. There is no freedom to delocalise the massive KK modes away from the IR brane, as was the case for the zero mode.
3.3 Yukawa Couplings and Fermion Masses
The SM fermions acquire masses via their Yukawa interactions with the Higgs in the process of EWSB. The effective 4D Yukawa matrices are then given by
(3.15) 
where is the fundamental 5D Yukawa coupling and is the Higgs shape function along the extra dimension, as given in (2.25).
We stress that together with from (2.25) the fermionic zero mode functions with respect to the warped metric, as given in (B.13), have to be used in order to determine the effective 4D Yukawa couplings in (3.15).
In the special case of an IR brane localised Higgs we obtain
(3.16) 
where are normalisation factors of the fermion shape functions on the IR brane. We note that in this case, up to , has a factorised form, as emphasised in [7]. In the more general case of the Higgs field propagating in the bulk this factorisation is broken, but only weakly for a large range of [67].
On the other hand we stress that in the presence of nondiagonal entries of , the factorisation must always be broken as not all entries of can be equal. Indeed this special choice of 5D Yukawa couplings would lead to two zero eigenvalues of the corresponding mass matrix. In addition some nondegeneracy between the entries of is needed to cure the “ problem” identified in [7]. Indeed this is completely analogous to the case of the FroggattNielsen flavour symmetry [68], where also a slight structure in the Yukawa coupling matrices is needed to obtain the correct size of . A detailed discussion of the close analogy between the FroggattNielsen mechanism and bulk fermions in RS models has been presented in [27].
4 Flavour Structure
4.1 Preliminaries
The flavour structure of this class of models is rather complicated, although as emphasised by Agashe et al. [8] in certain approximations it is quite simple. For the time being we will however not make any approximations.
The procedure to find all interactions including flavour violating ones is basically the following:
Step 1
We begin with the interaction terms in and in (A.1) in terms of flavour eigenstates for fermions and in the gauge eigenbasis for the gauge bosons.
To this end we start with the fundamental 5D interactions and then perform the KK decomposition as described in Appendix B. We thus obtain effective 4D couplings that are nonlocal quantities along the extra dimension, resulting from the overlap of the gauge boson and fermion shape functions. Schematically the interactions between different KK levels are given by
(4.1) 
Note that only fermions within the same gauge multiplet are coupled to each other in this way, so that their bulk mass parameters are necessarily equal. In the Feynman rules collected in Appendix D these overlap integrals appear as , and .
As the gauge boson zero mode has a flat shape function, the coupling for equal fermion KK levels to the gauge boson zero mode reduces to the 4D gauge coupling , while the integrals with vanish due to the normalisation of the fermion shape function. In the same way, couplings of different fermionic KK levels to the gauge boson zero mode vanish due to the orthogonality of the fermion shape functions. However, due to the effects of EWSB, the EW gauge boson zero mode mixes with its KK modes, so that eventually flavour nonuniversalities in the couplings of the SM weak gauge bosons and nonzero couplings between various KK levels will arise.
A different treatment of the effects of EWSB on the gauge boson zero modes has been discussed in [58, 59, 60]. In their description the Higgs VEV is inserted already at the level of the 5D equations of motion, which leads to a distortion of the gauge boson zero mode in the vicinity of the IR brane. On the other hand no mixing between the various KK levels appears. In Appendix C we review both approaches in detail, study their advantages and shortcomings and show that both interpretations are indeed physically equivalent. A similar independent discussion has been presented in [63].
Step 2
Next we have to consider the mass matrices of gauge bosons and fermions. Here the new aspects relative to the SM are

in the gauge sector:

The extended gauge group leads to the presence of additional (heavy) gauge bosons.

In addition, also the heavy KK modes of the SM gauge bosons, including also gluons and the photon, are present.

EWSB induces mixing of the SM zero modes with the additional heavy KK modes of the same electric charge.


in the fermion sector:

Also the heavy KK modes of the SM fermions have to be considered.

Again mixing of the SM zero modes with the heavy KK states is induced.
Step 3
Finally all interactions have to be rewritten in terms of mass eigenstates for gauge bosons and fermions. Therefore the corresponding mass matrices need to be diagonalised. Note that mixing takes place not only between different flavours, but also between different KK levels^{4}^{4}4This should be contrasted with models with flat extra dimensions where the presence of KK parity has eliminated such mixing. For a recent attempt to introduce KK parity in the RS framework, see [69].. As the new physics scale is experimentally constrained to be , within a good approximation it is sufficient to consider only the contributions of modes and neglect all higher KK levels. Therefore in what follows we restrict our discussion to this simplified case. The generalisation of our formulae to include also higher KK modes is then straightforward.
4.2 Quark Mass Matrices
The transformation to mass eigenstates in the gauge sector is performed in Section 2 and Appendix C. The goal of the present section is to construct and diagonalise the mass matrices for the quark fields given in (3.1)–(3.6). To this end we will only consider zero modes and the lowest () KK modes. As there are only few fields among the ones in (3.1)–(3.6) that have zero modes, we will assign to them the superscript . For the excited KK modes we will just use the notation of (3.1)–(3.6), making the index implicit.
We will have to deal with three mass matrices corresponding to the electric charges , and . To this end we group the fermion modes into the following vectors:
For the charge mass matrix we have
(4.2)  
(4.3) 
where the flavour index runs over the three quark generations. Thus we deal with 9dimensional vectors. Note that in this sector only massive excited KK states are present.
For the charge mass matrix the corresponding vectors read
(4.4)  
(4.5) 
Here the first components are zero modes, and so that we really deal with 18dimensional vectors.
The charge vectors read
(4.6)  
(4.7) 
Again the first entries are zero modes, the remaining ones massive KK modes, and , so that in this case a 12dimensional vector is obtained.
In order to construct the mass matrices let us briefly recall certain properties, known already from numerous studies in the literature:

We have three bulk mass matrices corresponding to the representations ( is the flavour index), respectively. Note that for a given multiplet with fixed flavour index all bulk mass parameters for different components of the multiplet are equal to each other.

In general, are arbitrary hermitian matrices, where corresponds to the multiplet . In the following we choose to work in the basis where they are real and diagonal, i. e. each of them is described by three real parameters , where is the flavour index. This can always be achieved by appropriate field redefinitions of the multiplets. Explicitly we then have:
(4.8) and similarly for and .

The allowed Yukawa couplings, giving mass to the fermion zero modes after EWSB, have to preserve the full gauge symmetry. The possible gauge invariant terms in the full 5D theory can be found in Appendix A.

The effective 4D Yukawa matrices will involve the fermion and Higgs shape functions. We will denote the fermionic ones by and , corresponding to the th and th component of and in (4.2)–(4.7), respectively, and is the Higgs shape function as given in (2.25).
Having at hand this information and restricting ourselves to for simplicity, we obtain the following effective 4D Yukawa couplings
(4.9) (4.10) (4.11) (4.12) Interestingly, the Yukawa coupling proportional to , connecting with and being thus responsible for the SM down quark Yukawa coupling, leads to mass terms not only for the charge quarks, but simultaneously also to mass terms for the and quarks. This is a direct consequence of and being placed in the adjoint representations of and , respectively, as seen in (3.3), (3.6).
On the other hand, the term proportional to , connecting with and being thus responsible for the SM up quark Yukawa coupling, contributes only to the mass matrix for the charge quarks.

Finally the fermionic KK masses, which can be obtained from solving the bulk equations of motion, have to be included in the mass matrices. Note that both the fermion shape function and the KK mass depend on the bulk mass parameter and on the BCs.
In terms of the mode vectors (4.2)–(4.7) we can write
(4.13)  
In order to distinguish zero modes from the KK fermions we will label the zero mode components of the vectors (4.2)–(4.7) by the index . Then the quark mass matrices read
(4.14) 
(4.15)  
(4.16) 
These three matrices have to be diagonalised via a biunitary transformation to find the quark mass eigenstates. Due to the large size of the mass matrices this diagonalisation has to be done numerically.
Let us make a few remarks:

The mass eigenstates of charge are all heavy.

In and the offdiagonal entries in the first column and row lead to mixing between light zero modes and heavy KK modes. This mixing will be suppressed by .

In the case of the Higgs field being confined exactly to the IR brane, only Yukawa couplings to those fermion modes are nonvanishing that obey a BC on the IR brane. In that case some of the entries in the above mass matrices in (4.14)–(4.16) vanish:
(4.17) (4.18) (4.19) As pointed out in [27] and discussed in detail in [70], this difference has profound implications on the size of flavour violating Higgs couplings.
We can then diagonalise the , and charge matrices by
(4.20)  