1 Acknowledgements

EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH (CERN)

CERN-PH-EP-2013-123 LHCb-PAPER-2013-030

Search for the lepton-flavour violating decays and

The LHCb collaboration111Authors are listed on the following pages.

A search for the lepton-flavour violating decays and is performed with a data sample, corresponding to an integrated luminosity of 1.0 of collisions at   TeV, collected by the LHCb experiment. The observed number of and candidates is consistent with background expectations. Upper limits on the branching fractions of both decays are determined to be and at 90% (95%) confidence level (C.L.). These limits are a factor of twenty lower than those set by previous experiments. Lower bounds on the Pati-Salam leptoquark masses are also calculated, and at 95% C.L., and are a factor of two higher than the previous bounds.

Published on Phys. Rev. Lett. 111, 141801 (2013)

© CERN on behalf of the LHCb collaboration, license CC-BY-3.0.

 

LHCb collaboration

R. Aaij, B. Adeva, M. Adinolfi, C. Adrover, A. Affolder, Z. Ajaltouni, J. Albrecht, F. Alessio, M. Alexander, S. Ali, G. Alkhazov, P. Alvarez Cartelle, A.A. Alves Jr, S. Amato, S. Amerio, Y. Amhis, L. Anderlini, J. Anderson, R. Andreassen, J.E. Andrews, R.B. Appleby, O. Aquines Gutierrez, F. Archilli, A. Artamonov, M. Artuso, E. Aslanides, G. Auriemma, M. Baalouch, S. Bachmann, J.J. Back, C. Baesso, V. Balagura, W. Baldini, R.J. Barlow, C. Barschel, S. Barsuk, W. Barter, Th. Bauer, A. Bay, J. Beddow, F. Bedeschi, I. Bediaga, S. Belogurov, K. Belous, I. Belyaev, E. Ben-Haim, G. Bencivenni, S. Benson, J. Benton, A. Berezhnoy, R. Bernet, M.-O. Bettler, M. van Beuzekom, A. Bien, S. Bifani, T. Bird, A. Bizzeti, P.M. Bjørnstad, T. Blake, F. Blanc, J. Blouw, S. Blusk, V. Bocci, A. Bondar, N. Bondar, W. Bonivento, S. Borghi, A. Borgia, T.J.V. Bowcock, E. Bowen, C. Bozzi, T. Brambach, J. van den Brand, J. Bressieux, D. Brett, M. Britsch, T. Britton, N.H. Brook, H. Brown, I. Burducea, A. Bursche, G. Busetto, J. Buytaert, S. Cadeddu, O. Callot, M. Calvi, M. Calvo Gomez, A. Camboni, P. Campana, D. Campora Perez, A. Carbone, G. Carboni, R. Cardinale, A. Cardini, H. Carranza-Mejia, L. Carson, K. Carvalho Akiba, G. Casse, L. Castillo Garcia, M. Cattaneo, Ch. Cauet, R. Cenci, M. Charles, Ph. Charpentier, P. Chen, N. Chiapolini, M. Chrzaszcz, K. Ciba, X. Cid Vidal, G. Ciezarek, P.E.L. Clarke, M. Clemencic, H.V. Cliff, J. Closier, C. Coca, V. Coco, J. Cogan, E. Cogneras, P. Collins, A. Comerma-Montells, A. Contu, A. Cook, M. Coombes, S. Coquereau, G. Corti, B. Couturier, G.A. Cowan, D.C. Craik, S. Cunliffe, R. Currie, C. D’Ambrosio, P. David, P.N.Y. David, A. Davis, I. De Bonis, K. De Bruyn, S. De Capua, M. De Cian, J.M. De Miranda, L. De Paula, W. De Silva, P. De Simone, D. Decamp, M. Deckenhoff, L. Del Buono, N. Déléage, D. Derkach, O. Deschamps, F. Dettori, A. Di Canto, H. Dijkstra, M. Dogaru, S. Donleavy, F. Dordei, A. Dosil Suárez, D. Dossett, A. Dovbnya, F. Dupertuis, P. Durante, R. Dzhelyadin, A. Dziurda, A. Dzyuba, S. Easo, U. Egede, V. Egorychev, S. Eidelman, D. van Eijk, S. Eisenhardt, U. Eitschberger, R. Ekelhof, L. Eklund, I. El Rifai, Ch. Elsasser, A. Falabella, C. Färber, G. Fardell, C. Farinelli, S. Farry, V. Fave, D. Ferguson, V. Fernandez Albor, F. Ferreira Rodrigues, M. Ferro-Luzzi, S. Filippov, M. Fiore, C. Fitzpatrick, M. Fontana, F. Fontanelli, R. Forty, O. Francisco, M. Frank, C. Frei, M. Frosini, S. Furcas, E. Furfaro, A. Gallas Torreira, D. Galli, M. Gandelman, P. Gandini, Y. Gao, J. Garofoli, P. Garosi, J. Garra Tico, L. Garrido, C. Gaspar, R. Gauld, E. Gersabeck, M. Gersabeck, T. Gershon, Ph. Ghez, V. Gibson, L. Giubega, V.V. Gligorov, C. Göbel, D. Golubkov, A. Golutvin, A. Gomes, H. Gordon, M. Grabalosa Gándara, R. Graciani Diaz, L.A. Granado Cardoso, E. Graugés, G. Graziani, A. Grecu, E. Greening, S. Gregson, P. Griffith, O. Grünberg, B. Gui, E. Gushchin, Yu. Guz, T. Gys, C. Hadjivasiliou, G. Haefeli, C. Haen, S.C. Haines, S. Hall, B. Hamilton, T. Hampson, S. Hansmann-Menzemer, N. Harnew, S.T. Harnew, J. Harrison, T. Hartmann, J. He, T. Head, V. Heijne, K. Hennessy, P. Henrard, J.A. Hernando Morata, E. van Herwijnen, A. Hicheur, E. Hicks, D. Hill, M. Hoballah, C. Hombach, P. Hopchev, W. Hulsbergen, P. Hunt, T. Huse, N. Hussain, D. Hutchcroft, D. Hynds, V. Iakovenko, M. Idzik, P. Ilten, R. Jacobsson, A. Jaeger, E. Jans, P. Jaton, A. Jawahery, F. Jing, M. John, D. Johnson, C.R. Jones, C. Joram, B. Jost, M. Kaballo, S. Kandybei, W. Kanso, M. Karacson, T.M. Karbach, I.R. Kenyon, T. Ketel, A. Keune, B. Khanji, O. Kochebina, I. Komarov, R.F. Koopman, P. Koppenburg, M. Korolev, A. Kozlinskiy, L. Kravchuk, K. Kreplin, M. Kreps, G. Krocker, P. Krokovny, F. Kruse, M. Kucharczyk, V. Kudryavtsev, T. Kvaratskheliya, V.N. La Thi, D. Lacarrere, G. Lafferty, A. Lai, D. Lambert, R.W. Lambert, E. Lanciotti, G. Lanfranchi, C. Langenbruch, T. Latham, C. Lazzeroni, R. Le Gac, J. van Leerdam, J.-P. Lees, R. Lefèvre, A. Leflat, J. Lefrançois, S. Leo, O. Leroy, T. Lesiak, B. Leverington, Y. Li, L. Li Gioi, M. Liles, R. Lindner, C. Linn, B. Liu, G. Liu, S. Lohn, I. Longstaff, J.H. Lopes, N. Lopez-March, H. Lu, D. Lucchesi, J. Luisier, H. Luo, F. Machefert, I.V. Machikhiliyan, F. Maciuc, O. Maev, S. Malde, G. Manca, G. Mancinelli, J. Maratas, U. Marconi, P. Marino, R. Märki, J. Marks, G. Martellotti, A. Martens, A. Martín Sánchez, M. Martinelli, D. Martinez Santos, D. Martins Tostes, A. Massafferri, R. Matev, Z. Mathe, C. Matteuzzi, E. Maurice, A. Mazurov, B. Mc Skelly, J. McCarthy, A. McNab, R. McNulty, B. Meadows, F. Meier, M. Meissner, M. Merk, D.A. Milanes, M.-N. Minard, J. Molina Rodriguez, S. Monteil, D. Moran, P. Morawski, A. Mordà, M.J. Morello, R. Mountain, I. Mous, F. Muheim, K. Müller, R. Muresan, B. Muryn, B. Muster, P. Naik, T. Nakada, R. Nandakumar, I. Nasteva, M. Needham, S. Neubert, N. Neufeld, A.D. Nguyen, T.D. Nguyen, C. Nguyen-Mau, M. Nicol, V. Niess, R. Niet, N. Nikitin, T. Nikodem, A. Nomerotski, A. Novoselov, A. Oblakowska-Mucha, V. Obraztsov, S. Oggero, S. Ogilvy, O. Okhrimenko, R. Oldeman, M. Orlandea, J.M. Otalora Goicochea, P. Owen, A. Oyanguren, B.K. Pal, A. Palano, M. Palutan, J. Panman, A. Papanestis, M. Pappagallo, C. Parkes, C.J. Parkinson, G. Passaleva, G.D. Patel, M. Patel, G.N. Patrick, C. Patrignani, C. Pavel-Nicorescu, A. Pazos Alvarez, A. Pellegrino, G. Penso, M. Pepe Altarelli, S. Perazzini, E. Perez Trigo, A. Pérez-Calero Yzquierdo, P. Perret, M. Perrin-Terrin, L. Pescatore, G. Pessina, K. Petridis, A. Petrolini, A. Phan, E. Picatoste Olloqui, B. Pietrzyk, T. Pilař, D. Pinci, S. Playfer, M. Plo Casasus, F. Polci, G. Polok, A. Poluektov, E. Polycarpo, A. Popov, D. Popov, B. Popovici, C. Potterat, A. Powell, J. Prisciandaro, A. Pritchard, C. Prouve, V. Pugatch, A. Puig Navarro, G. Punzi, W. Qian, J.H. Rademacker, B. Rakotomiaramanana, M.S. Rangel, I. Raniuk, N. Rauschmayr, G. Raven, S. Redford, M.M. Reid, A.C. dos Reis, S. Ricciardi, A. Richards, K. Rinnert, V. Rives Molina, D.A. Roa Romero, P. Robbe, D.A. Roberts, E. Rodrigues, P. Rodriguez Perez, S. Roiser, V. Romanovsky, A. Romero Vidal, J. Rouvinet, T. Ruf, F. Ruffini, H. Ruiz, P. Ruiz Valls, G. Sabatino, J.J. Saborido Silva, N. Sagidova, P. Sail, B. Saitta, V. Salustino Guimaraes, C. Salzmann, B. Sanmartin Sedes, M. Sannino, R. Santacesaria, C. Santamarina Rios, E. Santovetti, M. Sapunov, A. Sarti, C. Satriano, A. Satta, M. Savrie, D. Savrina, P. Schaack, M. Schiller, H. Schindler, M. Schlupp, M. Schmelling, B. Schmidt, O. Schneider, A. Schopper, M.-H. Schune, R. Schwemmer, B. Sciascia, A. Sciubba, M. Seco, A. Semennikov, K. Senderowska, I. Sepp, N. Serra, J. Serrano, P. Seyfert, M. Shapkin, I. Shapoval, P. Shatalov, Y. Shcheglov, T. Shears, L. Shekhtman, O. Shevchenko, V. Shevchenko, A. Shires, R. Silva Coutinho, M. Sirendi, T. Skwarnicki, N.A. Smith, E. Smith, J. Smith, M. Smith, M.D. Sokoloff, F.J.P. Soler, F. Soomro, D. Souza, B. Souza De Paula, B. Spaan, A. Sparkes, P. Spradlin, F. Stagni, S. Stahl, O. Steinkamp, S. Stevenson, S. Stoica, S. Stone, B. Storaci, M. Straticiuc, U. Straumann, V.K. Subbiah, L. Sun, S. Swientek, V. Syropoulos, M. Szczekowski, P. Szczypka, T. Szumlak, S. T’Jampens, M. Teklishyn, E. Teodorescu, F. Teubert, C. Thomas, E. Thomas, J. van Tilburg, V. Tisserand, M. Tobin, S. Tolk, D. Tonelli, S. Topp-Joergensen, N. Torr, E. Tournefier, S. Tourneur, M.T. Tran, M. Tresch, A. Tsaregorodtsev, P. Tsopelas, N. Tuning, M. Ubeda Garcia, A. Ukleja, D. Urner, A. Ustyuzhanin, U. Uwer, V. Vagnoni, G. Valenti, A. Vallier, M. Van Dijk, R. Vazquez Gomez, P. Vazquez Regueiro, C. Vázquez Sierra, S. Vecchi, J.J. Velthuis, M. Veltri, G. Veneziano, M. Vesterinen, B. Viaud, D. Vieira, X. Vilasis-Cardona, A. Vollhardt, D. Volyanskyy, D. Voong, A. Vorobyev, V. Vorobyev, C. Voß, H. Voss, R. Waldi, C. Wallace, R. Wallace, S. Wandernoth, J. Wang, D.R. Ward, N.K. Watson, A.D. Webber, D. Websdale, M. Whitehead, J. Wicht, J. Wiechczynski, D. Wiedner, L. Wiggers, G. Wilkinson, M.P. Williams, M. Williams, F.F. Wilson, J. Wimberley, J. Wishahi, M. Witek, S.A. Wotton, S. Wright, S. Wu, K. Wyllie, Y. Xie, Z. Xing, Z. Yang, R. Young, X. Yuan, O. Yushchenko, M. Zangoli, M. Zavertyaev, F. Zhang, L. Zhang, W.C. Zhang, Y. Zhang, A. Zhelezov, A. Zhokhov, L. Zhong, A. Zvyagin.

Centro Brasileiro de Pesquisas Físicas (CBPF), Rio de Janeiro, Brazil

Universidade Federal do Rio de Janeiro (UFRJ), Rio de Janeiro, Brazil

Center for High Energy Physics, Tsinghua University, Beijing, China

LAPP, Université de Savoie, CNRS/IN2P3, Annecy-Le-Vieux, France

Clermont Université, Université Blaise Pascal, CNRS/IN2P3, LPC, Clermont-Ferrand, France

CPPM, Aix-Marseille Université, CNRS/IN2P3, Marseille, France

LAL, Université Paris-Sud, CNRS/IN2P3, Orsay, France

LPNHE, Université Pierre et Marie Curie, Université Paris Diderot, CNRS/IN2P3, Paris, France

Fakultät Physik, Technische Universität Dortmund, Dortmund, Germany

Max-Planck-Institut für Kernphysik (MPIK), Heidelberg, Germany

Physikalisches Institut, Ruprecht-Karls-Universität Heidelberg, Heidelberg, Germany

School of Physics, University College Dublin, Dublin, Ireland

Sezione INFN di Bari, Bari, Italy

Sezione INFN di Bologna, Bologna, Italy

Sezione INFN di Cagliari, Cagliari, Italy

Sezione INFN di Ferrara, Ferrara, Italy

Sezione INFN di Firenze, Firenze, Italy

Laboratori Nazionali dell’INFN di Frascati, Frascati, Italy

Sezione INFN di Genova, Genova, Italy

Sezione INFN di Milano Bicocca, Milano, Italy

Sezione INFN di Padova, Padova, Italy

Sezione INFN di Pisa, Pisa, Italy

Sezione INFN di Roma Tor Vergata, Roma, Italy

Sezione INFN di Roma La Sapienza, Roma, Italy

Henryk Niewodniczanski Institute of Nuclear Physics Polish Academy of Sciences, Kraków, Poland

AGH - University of Science and Technology, Faculty of Physics and Applied Computer Science, Kraków, Poland

National Center for Nuclear Research (NCBJ), Warsaw, Poland

Horia Hulubei National Institute of Physics and Nuclear Engineering, Bucharest-Magurele, Romania

Petersburg Nuclear Physics Institute (PNPI), Gatchina, Russia

Institute of Theoretical and Experimental Physics (ITEP), Moscow, Russia

Institute of Nuclear Physics, Moscow State University (SINP MSU), Moscow, Russia

Institute for Nuclear Research of the Russian Academy of Sciences (INR RAN), Moscow, Russia

Budker Institute of Nuclear Physics (SB RAS) and Novosibirsk State University, Novosibirsk, Russia

Institute for High Energy Physics (IHEP), Protvino, Russia

Universitat de Barcelona, Barcelona, Spain

Universidad de Santiago de Compostela, Santiago de Compostela, Spain

European Organization for Nuclear Research (CERN), Geneva, Switzerland

Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland

Physik-Institut, Universität Zürich, Zürich, Switzerland

Nikhef National Institute for Subatomic Physics, Amsterdam, The Netherlands

Nikhef National Institute for Subatomic Physics and VU University Amsterdam, Amsterdam, The Netherlands

NSC Kharkiv Institute of Physics and Technology (NSC KIPT), Kharkiv, Ukraine

Institute for Nuclear Research of the National Academy of Sciences (KINR), Kyiv, Ukraine

University of Birmingham, Birmingham, United Kingdom

H.H. Wills Physics Laboratory, University of Bristol, Bristol, United Kingdom

Cavendish Laboratory, University of Cambridge, Cambridge, United Kingdom

Department of Physics, University of Warwick, Coventry, United Kingdom

STFC Rutherford Appleton Laboratory, Didcot, United Kingdom

School of Physics and Astronomy, University of Edinburgh, Edinburgh, United Kingdom

School of Physics and Astronomy, University of Glasgow, Glasgow, United Kingdom

Oliver Lodge Laboratory, University of Liverpool, Liverpool, United Kingdom

Imperial College London, London, United Kingdom

School of Physics and Astronomy, University of Manchester, Manchester, United Kingdom

Department of Physics, University of Oxford, Oxford, United Kingdom

Massachusetts Institute of Technology, Cambridge, MA, United States

University of Cincinnati, Cincinnati, OH, United States

University of Maryland, College Park, MD, United States

Syracuse University, Syracuse, NY, United States

Pontifícia Universidade Católica do Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil, associated to

Institut für Physik, Universität Rostock, Rostock, Germany, associated to

P.N. Lebedev Physical Institute, Russian Academy of Science (LPI RAS), Moscow, Russia

Università di Bari, Bari, Italy

Università di Bologna, Bologna, Italy

Università di Cagliari, Cagliari, Italy

Università di Ferrara, Ferrara, Italy

Università di Firenze, Firenze, Italy

Università di Urbino, Urbino, Italy

Università di Modena e Reggio Emilia, Modena, Italy

Università di Genova, Genova, Italy

Università di Milano Bicocca, Milano, Italy

Università di Roma Tor Vergata, Roma, Italy

Università di Roma La Sapienza, Roma, Italy

Università della Basilicata, Potenza, Italy

LIFAELS, La Salle, Universitat Ramon Llull, Barcelona, Spain

Hanoi University of Science, Hanoi, Viet Nam

Institute of Physics and Technology, Moscow, Russia

Università di Padova, Padova, Italy

Università di Pisa, Pisa, Italy

Scuola Normale Superiore, Pisa, Italy

Rare decays that are forbidden in the Standard Model (SM) probe potential contributions from new processes and particles at a scale beyond the reach of direct searches. The decays and and their charged conjugate processes111Inclusion of charge conjugate processes are implied throughout this Letter. are forbidden within the SM, in which lepton flavour is conserved. These decays are allowed in some scenarios beyond the SM that include models with heavy singlet Dirac neutrinos [1], supersymmetric models [2] and the Pati-Salam model [3]. The latter predicts a new interaction to mediate transitions between leptons and quarks via exchange of spin1 gauge bosons, called Pati-Salam leptoquarks (LQ), that carry both colour and lepton quantum numbers.

Current limits from ATLAS [4, 5, 6] and CMS [7, 8, 9] on the masses of first, second or third generation leptoquarks are in the range  , depending on the value of the couplings and the decay channel. These leptoquarks arise from a coupling between a quark and lepton of the same generation. The decays and can be mediated by other leptoquarks which couple leptons and quarks that are not necessarily from the same generation [10, 11], such as when the lepton couples to a first or second quark generation.

The previous best upper limits on the branching fraction of these decays come from the CDF collaboration [12], and at 90% (95%) confidence level (C.L.). These limits correspond to bounds on the masses of the corresponding Pati-Salam leptoquarks of and at 90 (95)% C.L. [12].

This Letter presents a search for the and lepton-flavour violating (LFV) decays performed with a data sample, corresponding to an integrated luminosity of 1.0  of collisions at   TeV, collected by the LHCb experiment in 2011 at the Large Hadron Collider. To avoid potential bias, events in the signal mass region were not examined until all analysis choices were finalized.

The LHCb detector is a single-arm forward spectrom- eter covering the pseudorapidity range , and is described in detail in Ref. [13]. Events were simulated for this analysis using the software described in Refs. [14, 15, 16, 17, 18, 19, 20].

The trigger [21] consists of a hardware stage (L0), based on information from the calorimeter and muon systems, followed by a software stage (HLT) that applies a full event reconstruction, and is split into two stages called HLT1 and HLT2. Candidate decays considered in this analysis must satisfy a hardware decision that requires the presence of a muon candidate with transverse momentum .

All tracks considered in the HLT1 are required to have . The muon track of the candidates is required to have and impact parameter,  mm. The HLT2 consists of exclusive, cut-based triggers for two-body decays, and inclusive multivariate [21, 22] -hadron triggers.

The decay is used as the normalization channel and () decays are used as a control channel, since both have the same event topology as the signal. The yield is computed from the yield of decays, and the fraction of in the sample, as described in Ref. [23]. In order to minimize the bias introduced by the trigger requirements, only candidates that are triggered independently of the presence of either of the two signal hadrons at L0 and HLT1 are considered.

The candidates that pass the trigger selection criteria are further required to have well identified electron and muon [24] candidates. The measured momenta of the electrons are corrected to account for loss of momentum by bremsstrahlung in the detector using the photon energy deposition in the electromagnetic calorimeter [25]. The signal candidates are required to be displaced with respect to any collision vertex (PV), and form a secondary vertex (SV) with per degree of freedom smaller than 9 and separated from the PV in the downstream direction by a flight distance significance greater than 15.

Only candidates with an impact parameter () less than 25 are considered. The of a candidate is defined as the difference between the of the PV reconstructed with and without the considered candidate. When more than one PV is reconstructed, that giving the smallest for the candidate is chosen. Only candidates with invariant mass in the range are kept for further analysis. The selection criteria for the and candidates are identical to those of the signal, apart from those used for particle identification.

A two-stage multivariate selection based on boosted decision trees (BDT) [26, 27] is applied to the candidates following the same strategy as Ref. [28]. The two multivariate discriminants are trained using simulated samples, for signal and for background (where can either be a or an and is any other set of particles), which is dominated by simultaneous semileptonic decays of both and hadrons within the same event.

The requirement on the first multivariate discriminant [28] removes 75 % of the background while retaining 93 % of signal, as determined from simulation using half of the available samples to train and the other half to evaluate the efficiencies. The same selection is applied to the normalization channel and the efficiencies of this requirement for the signal and normalization channel are equal within 1.2 %, as determined from simulation.

The surviving background mainly comprises random combinations of electrons and muons from semileptonic decays. In total 5766 electron-muon pairs pass the trigger, the offline selection and the first multivariate discriminant requirements. The selected candidates are classified in a binned two-dimensional space formed by the electron-muon invariant mass and the output of a second BDT, for which nine variables are employed [28]. The BDT output is independent of the invariant mass for signal inside the search window. The output is transformed such that the signal is approximately uniformly distributed between zero and one, while the background peaks at zero.

The probability for a signal event to have a given BDT value is obtained from data using the sample [29, 30]. Simulated samples of and decays have been used to check that the distributions of the variables entering in the BDT that do not depend on the bremsstrahlung radiation are in good agreement. Corrections to the BDT shape due to the presence of the radiation emitted by the electron of the decays have been evaluated using simulation. The number of signal events in each BDT bin is determined by fitting the invariant mass distribution. The systematic uncertainty on the signal BDT probability distribution function is taken to be the maximum spread in the fractions of yields going into each bin, obtained by fitting the same dataset with different signal and background fit models. Corrections are applied to the BDT shape in order to take into account the effect of the different trigger requirements used for the signal and the control sample.

The invariant mass line shape of the signal events is described by a Crystal Ball function (CB) [31] with two tails, left and right, defined by two parameters each. The values of the parameters depend on the momentum resolution, the momentum scale and the amount of bremsstrahlung radiation recovered.

The signal shape parameters are obtained from simulation, but need to be reweighted to account for their dependency on the event multiplicity, which affects the amount of bremsstrahlung radiation recovered and differs between data and simulation. We use the number of hits in the scintillating pad detector () as a measure of the event multiplicity. The distribution of for signal candidates is obtained from a data sample, which is selected with the same trigger conditions as the signal, ensuring a similar distribution of . The signal mass shape parameters are determined by reweighting the simulated events with the distribution measured in the sample.

This reweighting technique is used also for a simulated sample and the reweighted parameters are then compared with those obtained with a sample in data. The difference between the mean values of the mass in data and simulation (+0.16%) is applied as a systematic shift to the peak values of the and invariant mass in simulation. A systematic uncertainty is added to the mass parameters when the differences in the values of the other mass parameters for the sample in data and SPD-reweighted simulation are larger than their statistical uncertainties.

The signal region, defined by the invariant mass window , retains and of the and signal decays, respectively. The systematic uncertainties on these fractions are evaluated with pseudo-experiments that fluctuate each parameter of the mass lineshape according to its uncertainty. The width of the corresponding fraction distribution is taken as the systematic uncertainty.

The and yields are translated into branching fractions according to

(1)

where is the number of signal events in the normalization channel and is determined from the total yield of the channel and the fraction of events in the inclusive sample. The systematic uncertainty is comparable to the statistical one and is dominated by the maximum spread in the yield obtained by fitting the same dataset with different fit models [29, 30]. The branching fraction of the normalization channel is  [32] and is the number of observed signal events. The factors and indicate the probabilities that a quark fragments into a or meson, respectively. We use measured in collision data at TeV [33]. The measured dependence of on the meson  [33] is found to be negligible for this analysis.

The efficiency for the signal (normalization) channel is the product of the reconstruction efficiency of the final state particles including the geometric detector acceptance, the selection efficiency and the trigger efficiency. The ratios of acceptance, reconstruction and selection efficiencies are computed with simulation. A systematic uncertainty is assigned to these ratios, to take into account the difference between the tracking efficiencies measured in data and predicted in simulation. Reweighting techniques are used to correct distributions in the simulation that do not match those from data, in particular for those variables that depend on . The trigger efficiency of L0 and HLT1 on signal decays is evaluated using data, while the HLT2 efficiency is evaluated in simulation after validation with control samples. The electron and muon identification efficiencies are evaluated from data using the and control samples. The two normalization factors and are determined to be and .

The BDT range is divided into eight bins with boundaries at and 1.0. The number of expected combinatorial background events in each BDT bin and the invariant mass signal region is determined from data by fitting to an exponential function events in the mass sidebands, defined by and . The invariant mass distributions of the selected candidates in BDT bins and the binned BDT distributions for the signals and the combinatorial background samples are available in Supplemental Material [34].

In the exponential function both the slope and the normalization are allowed to vary. The systematic uncertainty on the estimated number of combinatorial background events in the signal regions is determined by fluctuating the number of events observed in the sidebands according to a Poisson distribution, and by varying the exponential slope according to its uncertainty. As a cross-check, two other models, the sum of two exponential functions and a single exponential fitted to the right sideband only, have been used and provide consistent background estimates inside the signal region.

The low-mass sideband and the signal region are potentially polluted by exclusive backgrounds. The background from and decays is evaluated assuming the branching fraction value from Ref. [35]. The decays , , , and (where or ) are potential backgrounds if the hadrons are misidentified as electrons or muons. The and branching fractions are taken from Ref. [32]. The branching fraction is taken from Ref. [36]. The theoretical estimates of the and branching fractions are taken from Refs. [37] and [38], respectively. We use the fragmentation fraction measured by LHCb [39] and account for its dependence.

The mass and BDT distributions of these background modes are evaluated from simulated samples, using the probabilities of misidentifying kaon, pion and proton as muon or electron as functions of momenta and transverse momenta, which are determined from and data samples. The yield of the peaking background in each BDT bin is obtained by multiplying the yields obtained by fitting the invariant mass distribution of an inclusive sample in BDT bins [29, 30] with the probabilities of misidentifying kaon, pion and proton as muon or electron as functions of momenta and transverse momenta, as determined from control samples. The mass lineshape of the peaking background is obtained from a simulated sample of doubly-misidentified events. Apart from , all background modes are normalized relative to the decay. We assume where is the fragmentation fraction.

The and the and modes are the dominant exclusive modes in the range BDT, where the combinatorial background is reduced by a factor according to simulation. These decay modes have an invariant mass distribution that is compatible with an exponential in the region [4.9-5.9] , and hence are taken into account by the exponential fit to the mass sidebands.

In the entire BDT and mass range (), doubly misidentified decays are expected, with lying in the signal mass interval of

For each BDT bin we count the number of candidates observed in the signal region, and compare to the expected number of signal and background events.

The systematic uncertainties in the background and signal predictions in each bin are computed by varying the normalization factor, and the mass and BDT shapes within their Gaussian uncertainties.

The results for the and decays are summarized in Table 1. In the high BDT range, the observed number of candidates is in agreement with the number of expected exclusive backgrounds in the signal region. The compatibility of the observed distribution of events with that expected for a given branching fraction hypothesis is computed with the method [40].

BDT bin 0.0 – 0.25 0.25 – 0.4 0.4 – 0.5 0.5 – 0.6 0.6 – 0.7 0.7 – 0.8 0.8 – 0.9 0.9 – 1.0
Expected bkg (from fit)
Expected bkg 0.670.12 0.470.09 0.400.08 0.370.06 0.450.08 0.490.08 0.570.09 0.540.12
Observed
Table 1: Expected background (bkg) from the fit to the data sidebands, and expected events, compared to the number of observed events in the mass signal region, in bins of BDT response.

The expected and observed values are shown in Fig. 1 for the and channels, as a function of the assumed branching fraction. The expected and measured limits for and at 90 % and 95 % C.L. are shown in Table 2. Note that since the same events are used to set limits for both and decays, the results are strongly correlated. The inclusion of systematic uncertainties increases the expected and upper limits by . The systematic uncertainties are dominated by the uncertainty in the interpolation of the background yields inside the signal region. The observed limits are below the expectation due to the lower than expected numbers of observed events in the fourth and last BDT bins.

Figure 1: as a function of the assumed branching fraction for (left) and (right) decays. The dashed lines are the medians of the expected  distributions if background only was observed. The yellow (green) area covers, at a given branching fraction, 34%(47.5%) of the expected distribution on each side of its median. The solid black curves are the observed . The upper limits at 90 % (95 %) C.L. are indicated by the dotted (solid) vertical lines in blue for the expectation and in red for the observation.
Mode Limit 90 % C.L. 95 % C.L.
Expected
Observed
Expected
Observed
Table 2: Expected (background only) and observed limits on the branching fractions.

In the framework of the Pati-Salam model, the relation linking the branching fractions and the leptoquark mass ([10] is