# LHC Signatures of Two-Higgs-Doublets

with Fourth Family

###### Abstract:

On-going Higgs searches in the light mass window are of vital importance for testing the Higgs mechanism and probing new physics beyond the standard model (SM). The latest ATLAS and CMS searches for the SM Higgs boson at the LHC (7 TeV) found some intriguing excesses of events in the channels () around the mass-range of GeV. We explore a possible explanation of the and signals from the light CP-odd Higgs or CP-even Higgs from the general two-Higgs-doublet model with fourth-family fermions. We demonstrate that by including invisible decays of the Higgs boson or to fourth-family neutrinos, the predicted and signals can explain the observed new signatures at the LHC, and will be further probed by the forthcoming LHC runs in 2012.

JHEP (2012), in Press, [ arXiv:1202.3072 ]

###### Contents

## 1 Introduction

The LHC searches for Higgs boson(s) in the light mass window
have vital importance for testing the Higgs mechanism [1]
and probing new physics beyond the SM. The most recent results from the LHC (7 TeV)
have constrained the light Higgs boson of the standard model (SM)
into the mass-range (115.5 GeV, 131 GeV)
by ATLAS [2] and
(115 GeV, 128 GeV) by CMS [3], at C.L.^{1}^{1}1From the
latest updates at the Moriond conference [4],
ATLAS further confined the allowed light SM Higgs mass ranges into
(117.5 GeV, 118.5 GeV) and (122.5 GeV, 129 GeV) at C.L.,
while CMS gave the improved Higgs mass limits of (114.4 GeV, 127.5 GeV).
In particular, the ATLAS observed
an intriguing excess of events for
a Higgs boson with mass close to [2].
The three most sensitive channels in this mass range,
, , and ,
contribute to the excess with local significances of
, , and , respectively.
If this would be confirmed by the upcoming LHC data in 2012, a Higgs boson of mass
around 126 GeV does call for new physics beyond the SM due to the
vacuum instability [5]. Furthermore, the observed
excess in the channel by ATLAS is also
higher than the expected signals of the pure SM Higgs boson (with the same mass) by a factor-2 [2], which again points to new physics.

In this work, we investigate a simple SM-extension as the new physics — the generic two-Higgs-doublet model (2HDM) with fourth-family SM fermions (4F2HDM). It contains the minimal extension in the SM Higgs sector with one more doublet and in the SM fermion sector with one more family. With such a truly simple addition, we study distinct new signatures of the light CP-odd Higgs or CP-even Higgs at the LHC, and analyze the implications for the latest ATLAS and CMS Higgs searches [2][3]. We consider the 4F2HDM in both type-I and type-II, with CP-conserving Higgs potential. Such 2HDMs contain four physical Higgs states with masses . Due to the additional contributions from heavy fourth-family quarks , the gluon-fusion production cross sections of at the LHC are generally much enhanced relative to in the SM, and thus may be easily excluded by the current LHC data. In the present study, we demonstrate that the invisible Higgs decays into the light fourth-family neutrinos, , can become the major channel, and play a key role to properly suppress rates for the consistency with the existing LHC data. Especially, we show that such a light Higgs boson or with mass around GeV can nicely explain the observed event excesses by ATLAS [2] and CMS [3].

## 2 Signals of CP-Odd in 4F2HDM with Invisible Decays

We start with the analysis of CP-odd Higgs boson . The general 2HDM allows to be the lightest Higgs boson for proper parameter space of the Higgs potential, unlike the minimal supersymmetric SM (MSSM) where the lightest Higgs boson is always . An explicit realization of such 2HDMs is given by the dynamical top-seesaw model [6], where the light mass of the composite pseudo-scalar is induced by the topcolor instanton effect [7] and thus can naturally serve as the lightest state in the Higgs spectrum. Since has no cubic gauge couplings at tree-level, it mainly decays into the SM fermion pairs and the final states (via triangular fermion-loops). So the decay channel could be important for detecting such a light boson at the LHC. However, it was recently found [8, 9] that a light in the presence of fourth-family is excluded due to the enhanced cross section and unsuppressed decay branching ratio of . We note that this exclusion holds only in certain parameter region. In the following, we will include the invisible decays for the fourth-family neutrinos being lighter than half of , and study the distinct new LHC signatures of the Higgs boson.

The Higgs potential of the general 2HDM contains two characteristic input parameters, the as the ratio of two Higgs vacuum expectation values (VEVs), and the mixing angle from diagonalizing the mass-matrix of neutral Higgs bosons . It was shown [10] that such 2HDM with fourth-family fermions is consistent with the electroweak precision constraints. Ref. [11] also found that within broad parameter regions, the 4F2HDM can satisfy the and mixing constraints. For the present study, we focus on two types of CP-conserving 2HDMs without tree-level flavor-changing neutral currents (FCNC) [12], the type-I and type-II 2HDMs including the fourth-family. By definition, the type-I 2HDM assigns the first Higgs doublet (with VEV ) to couple with all fermions via Yukawa interactions and generate their masses, but the second Higgs doublet (with VEV ) does not. The type-II 2HDM has couple to all up-type fermions and to all down-type fermions. The most general Yukawa interactions for the pseudo-scalar in the 4F2HDM can be expressed as,

(1) |

where the couplings in the 4F2HDM-I and -II are summarized in Table 1.

4F2HDM-I | 4F2HDM-II | |
---|---|---|

The major production channel of at the LHC is the gluon-fusion process, and its cross section differs from that of the SM Higgs boson (coupled to three families of SM fermions) through the ratio,

(2) |

Here the form factors for the CP-even and CP-odd Higgs bosons take the forms [13],

(3) | |||

(4) |

with . Notice that the ratio of the on-shell production cross sections (2) is clearly independent of the center-of-mass energy of the LHC. This is also true for the ratio of the corresponding signal event numbers, as the integrated luminosity is the same for both cross sections. Hence our predicted ratio of signals for either production cross sections or number of events should also apply to the forthcoming LHC runs with higher collision energies and/or higher luminosities [14]. Then, we compute the ratio (2) for the inputs and in Fig. 1. For larger vlaues, the ratio (2) for 4F2HDM-II receives an enhancement from fourth-family quark . Thus the production cross section is more enhanced for large relative to that of the SM Higgs boson. On the other hand, all type-I Yukawa couplings are controlled by an overall factor as shown in Table 1. So the fourth-family quarks give contributions proportional to a uniform factor . Obviously, the fourth-family corrections to the gluon-fusion cross section is enhanced by in 4F2HDM-II while suppressed by in 4F2HDM-I for . Fig. 1 shows that for , the production is always enhanced in 4F2HDM-II, and the enhancement factor is about for in the mass-range GeV. In contrast, the production in 4F2HDM-I is moderately enhanced by a factor of for and GeV, which is much lower than that of 4F2HDM-II with the same . Due the opposite signs between the up-type and the down-type Yukawa couplings and of 4F2HDM-I (Table 1), a cancellation appears between their contributions to the ratio (2). This cancellation becomes maximal when the two heavy quarks are degenerate. So the cross sections in (2) are dominated by the third-family top-quark-loop and the inequality for a given Higgs mass determines the final enhancement of the ratio (2) for , as shown in Fig. 1 for 4F2HDM-I. We also see that for a larger , such as , the production in the 4F2HDM-I is suppressed by about a factor-10 relative to that of the in the SM3.

In general, the type-II Higgs sector is more nontrivial and interesting than the type-I, it is also well motivated for the fermion mass generations. In the natural parameter-space of , it is very challenging to make the 4F2HDM-II safe from the LHC constraints as noted before [8][9]. We have to sufficiently reduce the signals by suppressing the relevant decay branching fractions of . For this purpose, we propose a new resolution by exploring the invisible decays of into light fourth-family neutrinos, .

Generally, the fourth-family neutrinos have both Dirac and Majorana mass-terms which form the seesaw mass-matrix,

(5) |

After the diagonalization into mass-eigenbasis , their mass-eigenvalues are determined by the two mass-parameters and ,

(6) |

with the mixing angle defined as,

(7) |

where must hold due to . The case of corresponds to , leading to two degenerate states of pure Dirac neutrinos. The limit of is unphysical since it gives . The LEP precision data on invisible decays constrain , while the naturalness requires Yukawa couplings to be of and thus the Dirac mass GeV. Hence, our parameter space for the mixing angle is confined into the range of . The fourth-family neutrino can be stable on the collider lifetime, and the current experimental lower limits on stable neutral heavy lepton mass is as low as GeV at 95% C.L., as inferred from the invisible width [16, 17]. Taking into account of the mixing between two Majorana neutrinos and , this bound may be further reduced to GeV [15]. Such light fourth-family neutrinos will open up new invisible decay channels for both and . The lower limit on the mass of fourth-family charged lepton is about GeV, as given by the LEP-II direct searches [18]. These limits show that the fourth-family neutrinos and leptons () can be much lighter than the fourth-family quarks . For short-lived with prompt decays of and , the current searches at the LHC (7 TeV) places the following lower bounds (95% C.L.), GeV from the CMS with [19] or GeV from the ATLAS with [20], and GeV [21]. Meanwhile, the latest analysis from the Tevatron searches [22] using an integrated luminosity of for both CDF and D0 places the lower mass limits, GeV and GeV at 95% C.L. For illustration in the following analysis, we will uniformly take a sample input of fourth-family fermion masses, GeV, unless specified otherwise.

For the present analysis, we will systematically explore the new decay channels of in the 4F2HDM, , as well as , when is heavier than twice of () or . The invisible decay widths of are computed at the tree-level,

(8) | |||||

(9) |

where the second channel (9) is open when . The fourth-family fermions also contribute to the loop-induced decay widths for as follows,

(10) | |||||

(11) |

(12) |

where and denote the electric charge and color-factor for each fermion species. Besides, , and with being the weak mixing angle. All decay widths in our analysis are computed by including the relevant NLO QCD corrections as in Ref. [23]. The form factor in (12) is given by

(13) |

with . Since the form factors and are positive for the fermionic contributions, all three decay widths in (10)-(12) are larger than in the SM3. Including the new invisible decay channels of with decay rates in (8)-(9) , it is possible to suppress all SM decay branching fractions for the low-mass range of . In Fig. 2, we show the decay branching ratios in a wide mass-range of GeV for the 4F2HDM-II. We take the sample inputs of , where corresponds to the case of being pure Dirac neutrino. Fig. 2 shows that the invisible decay can dominate over all other channels for , while the and channels become dominant for . In particular, the diphoton channel can be suppressed by a factor of for , as compared to the diphoton branching fraction of the SM Higgs boson in the same mass range. Combining this with the enhanced cross sections in Fig. 1, we see that the new invisible decays of play a key role to bring down the signals for being consistent with the current LHC searches. Moreover, has vanishing cubic couplings with gauge bosons and thus no final states will be produced. It is clear that the current limits on the mass-range of will be much weaker than that of the conventional SM Higgs boson (mentioned at the beginning of Sec. 1).

Motivated by the latest ATLAS data [2], we focus on the case of a light with mass GeV. The invisible decay mode is kinematically allowed for , where the lower limit comes from the LEP constraints [16, 17]. For the invisible decay rates (8)-(9), we have included both Dirac and Majorana neutrino masses. In the pure Dirac-mass limit , the two fourth-family neutrinos become degenerate and thus their decay rates are equal, .

In Fig. 3(a), we analyze the decay branching fractions of versus the mixing angle of the fourth-family neutrinos. It shows that the invisible decay channel always dominates over all other channels for the full allowed range . The branching ratios for all other SM decay channels are maximized around , at which the second invisible decay channel is kinematically forbidden. In Fig. 3(b), we further analyze various decay branching fractions versus for the 4F2HDM-II. The invisible decay branching ratio Br[] is maximized for small . When gets larger, it gets reduced and no longer dominates over other channels; this could potentially cause too large signals at the LHC for the 4F2HDM-II. Such a danger is absent for the 4F2HDM-I, where all the partial decay widths are suppressed by due to . Besides, in the 4F2HDM-I the production cross section of gets suppressed for larger , as shown in Fig. 1.

Now we are ready to evaluate the signal predictions of for in the 4F2HDM (type-I and type-II), and then derive the ratio , where the denominator is the corresponding signals in the SM3 with the same input of Higgs mass as . We present our results in Fig. 4 for the LHC (7 TeV). It shows that for the 4F2HDM-I, the predictions are always significantly below that of the SM3. For the 4F2HDM-II with , we find that the signals can be moderately larger than that of the SM3 in the parameter region, with [Fig. 4(a)], or with [Fig. 4(b)]. But, larger values of in Fig. 4(a) would cause too much excess of signals, and are excluded by the current data. With the data set, ATLAS collaboration observed excess of events at the invariant-mass GeV, while the expected SM Higgs signal is above the SM backgrounds, which is about a factor-2 smaller than what ATLAS observed [2]. The signal-reduction-rate due to various cuts and detection efficiencies should be roughly the same for both the SM Higgs boson and the CP-odd boson. When the predicted ratio in the 4F2HDM-II, the signals can nicely explain the excess of ATLAS observation at GeV. For instance, this is realized at in Fig. 4(a).

As shown by Figs. 2-3, the invisible decays can dominate over all other channels in the relevant parameter regions. The suppression on the fermionic decay branching ratios (such as ) appears moderate in comparison with the SM3 case. Nevertheless, the Higgs searches in the and decay modes [3] are made through the vector boson associated production and the vector boson fusion processes, respectively, which receive no new enhancement from the fourth-family fermions. This is consistent with the present observations of ATLAS [2] and CMS [3], which found no excess from the and final states. Similar reasoning also holds for the detection of the CP-even Higgs boson via the and channels (cf. Sec. 3). Furthermore, the CMS detector showed no new signals in the final states, and ATLAS analysis only indicated a smaller excess in the events. It is very likely that the channels contains only the SM backgrounds. If so, this is again consistent with our analysis of the Higgs boson, since the CP-odd has no tree-level gauge couplings with and thus the decays are forbidden.

Note that in each plot of Fig. 4, the two solid curves correspond to GeV, and the two dashed curves to GeV. We do not show a curve for GeV since it almost overlaps with that of GeV. So the parameter space between the two adjacent curves in each set (either red or blue) in Fig. 4 essentially represent that of the mass-range GeV. From Fig. 4, we see that should the present ATLAS excess at GeV be disconfirmed by this summer with more LHC data, our 4F2HDM-II can predict new Higgs signals in other values around GeV, either above or below the SM3 Higgs rates. This will be further probed by the LHC Higgs searches.

## 3 Signals of CP-Even in 4F2HDM with Invisible Decays

In this section, we turn to the analysis of the CP-even Higgs boson in the 4F2HDM, and study the impacts of the invisible decays on the LHC discovery. Due to the mixing between the two CP-even neutral states, we have the mixing angle as a new input parameter. Unlike , the CP-even Higgs boson also has additional decay channels of at tree-level. We will present a benchmark model for the 4F2HDM, and analyze the production and decays of at the LHC. We further compare our predictions to that of the SM Higgs coupled with four families of fermions (SM4), by including the invisible decay channel of .

4F2HDM-I | 4F2HDM-II | |
---|---|---|

In the 4F2HDM, the analysis of production and decays of are more complicated than , due to the additional decay channels in the and final states, as well as the mixing parameter associated with two CP-even states . The Yukawa interactions of can be generally expressed as follows,

(14) |

where the Yukawa couplings for the 4F2HDM-I and 4F2HDM-II are summarized in Table 2. The production via the gluon-fusion process receives new contributions from the fourth-family quarks , which are enhanced by the Yukawa couplings of relative to that of the SM top quark (). We compute the ratio of production cross sections between the 4F2HDM and SM3 with the same mass of ,

(15) |

which is found to be generally larger than unity. In Fig. 5, we present the enhancement factors (15) for 4F2HDM-I (blue curve) and 4F2HDM-II (red curve) with the sample input . The enhancement (15) is moderate since in (15) can be smaller than one, as compared to the SM4 with . For the SM4, we see from the purple curve in Fig. 5, holds in the limit of light Higgs mass with the loop-contributions from the heavy quarks.

We analyze the impact of fourth-family fermions on the Higgs boson decays. For relatively light fourth-family neutrinos and leptons, new decay channels of can be open, in addition to the conventional SM decay modes whose partial widths have rescaling factors and for the fermionic and final states, respectively. Therefore, we shall rewrite the loop-induced decay rates in terms of the modified couplings for both bosonic and fermionic contributions,

(16) |

(17) | |||||

(18) | |||||

with the form factors,

(19a) | |||||

(19b) | |||||

(19c) | |||||

(19d) | |||||

(19e) | |||||

(19f) |

The charged Higgs loops may also contribute to the (17) and (18). For our illustration, we consider the large limit where the contributions are negligible. Such large limit is also fairly reasonable from the flavor physics constraints, including the leptonic decay of mesons , loop-induced transitions, and the mass difference as measured in the mixing [25, 26]. For lighter , the inclusion of charged Higgs loop will not affect our physical conclusion. In Fig. 6, we presented a sample of decay branching fractions for as a function of its mass in the 4F2HDM-II, by including the new invisible decay modes. It clearly shows that the invisible decays can suppress the other decay channels in the light mass region of . For , they no longer dominate because of . From Fig. 5-6, we see that for a light with mass , its production and decays in the 4F2HDM are very different from that of the SM Higgs boson due to the fourth-family quark contributions and the new channels of invisible decays.

In Fig. 7(a), we present the decay branching fractions of as a function of with , while in Fig. 7(b) we display Br as a function of with . We find that Br is sensitive to , and for the invisible decays dominate over all other SM channels in the full range of .

In Fig. 8-9, we present the predicted for the processes and in both 4F2HDM-I and 4F2HDM-II. In particular, we show the ratio , for the comparison to that of the SM Higgs boson, with the same mass GeV. In Fig. 8, we plot the ratios for the 4F2HDM-I and 4F2HDM-II as functions of the neutrino mixing parameter , with a sample input of . The predictions of the 4F2HDM-I are generally suppressed in comparison with the SM3, thus they cannot be observed from the current LHC data. To detect in the 4F2HDM-I thus requires higher integrated luminosities at the LHC. For predictions of the 4F2HDM-II, Fig. 8 shows interesting excess of signals above that of the SM Higgs boson for both and channels in the parameter range (with ), where the signals are significantly higher than the signals. We note that such 4F2HDM-II model with is quite generic for the dynamical fourth-family models [24]. Combined with the invisible decay channels, this can nicely explain why ATLAS experiment [2] has detected sizable excess of events in the mode but not the and final states. Fig. 9(a)-(b) depict these signal ratios as functions of for the neutrino mixing parameter and , respectively. We find that the 4F2HDM-II always predicts larger signals than the SM3 in the channel for , while the signals are mainly suppressed except for (with ).

For comparison with our above 4F2HDM studies, we also analyze the signals in the and channels from the one-Higgs-doublet SM including fourth-family (SM4), with relatively light . Ref. [27] showed signals from in the SM4 without including invisible decays, while the effect of invisible decays for the SM4 Higgs was discussed in [28] for the LEP searches and in [15, 29] for the LHC searches. In Fig. 10, we present the predicted signal ratios of in the processes and as functions of Higgs mass for the SM4. For comparison, we assign the fourth-family neutrino mass, GeV in Fig. 10(a) and GeV in Fig. 10(b), respectively. From these plots, we see that the signals are always much more suppressed than the signals. For the final states, besides the overall suppression from invisible decays , the fourth-family fermions further suppress the decay width of due to the enhanced fermion-loop contributions that cancel against the -loop in (17). Hence, for the Higgs detection in the SM4, the LHC should observe significantly larger signals than the signals; this is just opposite to the most recent ATLAS and CMS observations [2][3]. Furthermore, Fig. 10(a) shows that the SM4 predictions for the light mass-range GeV are generally lower than that of the conventional SM3 through all three decay channels of , and . Hence, higher integrated luminosities at the LHC are required for its detection. For Fig. 10(b) with a larger fourth-neutrino mass GeV, this window shifts to GeV. Fig. 10(b) shows that the SM Higgs boson with GeV or GeV is clearly excluded by the current LHC data due to excessive signals in the final states. If the recent event excesses around mass-values of GeV at the LHC (7 TeV) [2, 3] are actually due to statistical fluctuations or other systematical errors, then the low Higgs-mass-ranges of the SM4, namely GeV in plot-(a) and GeV in plot-(b), are still viable and will be further probed at the LHC with higher luminosities.

## 4 Conclusions

The on-going LHC Higgs searches for the light mass window ( GeV) are crucial for testing the Higgs mechanism and probing new physics beyond the SM. In this work, we studied the new signatures of a light CP-odd Higgs or CP-even Higgs in the and channels () at the LHC (7 TeV), as predicted by the two-Higgs-doublet-model with the fourth-family fermions (4F2HDM). By including the invisible decays of Higgs boson or into fourth-family neutrinos , we demonstrated that the predicted and signals can explain the recently observed excesses of events in ATLAS [2] and CMS [3] detectors. Due to the absence of cubic gauge-couplings -- , the decay channel becomes unique for discovering in the light mass-range GeV. Although the fourth-family quark-loops significantly enhance the production cross section of the gluon-fusion process relative to that of in the conventional three-family SM (SM3) as in Fig. 1, the invisible decay modes can properly suppress the branching fraction (Figs. 2-3) and make the signals mildly exceed that of the SM3 (Fig. 4). Hence, we found that for our 4F2HDM-II (with generic type-II Higgs sector), a light with mass GeV can nicely explain the excess of signals at the LHC [2, 3]. At the same time, the Higgs boson gives no signal for channels. Note that the latest ATLAS search showed lower excesses in modes and the CMS analysis found no excess in the same channel. If a light Higgs boson indeed exists, more LHC data in 2012 will pin down the possible signals in both and