Energy Flow Networks: Deep Sets for Particle Jets
Abstract
A key question for machine learning approaches in particle physics is how to best represent and learn from collider events. As an event is intrinsically a variablelength unordered set of particles, we build upon recent machine learning efforts to learn directly from sets of features. Adapting and specializing the “Deep Sets” framework to particle physics, we introduce Energy Flow Networks, which respect infrared and collinear safety by construction. We also develop Particle Flow Networks, which allow for general energy dependence and the inclusion of additional particlelevel information such as charge and flavor. These networks feature a perparticle internal (latent) representation, and summing over all particles yields an overall eventlevel latent representation. We show how this latent space decomposition unifies existing event representations based on detector images and radiation moments. To demonstrate the power and simplicity of this setbased approach, we apply these networks to the collider task of discriminating quark jets from gluon jets, finding similar or improved performance compared to existing methods. We also show how the learned event representation can be directly visualized, providing insight into the inner workings of the model. These architectures lend themselves to efficiently processing and analyzing events for a wide variety of tasks at the Large Hadron Collider. Implementations and examples of our architectures are available online in our EnergyFlow package.
77 Massachusetts Avenue, Cambridge, MA 02139, U.S.A.^{†}^{†}institutetext: Department of Physics, Harvard University,
17 Oxford Street, Cambridge, MA 02138, U.S.A.\preprint
MIT–CTP 5064
1 Introduction
Collisions at accelerators like the Large Hadron Collider (LHC) produce multitudes of particles. Particles are the fundamental objects of interest in collider physics and provide an interface between theoretical calculations and experimental measurements, often reconstructed experimentally via “particle flow” algorithms Beaudette:2014cea (); Sirunyan:2017ulk (); Aaboud:2017aca (). Analyses of collider data rely on observables to distill these complicated multiparticle events and capture essential aspects of the underlying physics. Because each collision event consists of a variablelength list of particles with no intrinsic ordering, collider observables must be sensibly defined as functions of sets of particles. In this paper, we develop a novel architecture for processing and learning from collider events in their natural setbased representation.
Recently, modern machine learning techniques have been used to achieve excellent performance on a variety of collider tasks by learning specalized functions of the events, which can be viewed as observables in their own right. For instance, hadronic jet classification has been thoroughly studied using lowlevel Cogan:2014oua (); deOliveira:2015xxd (); Baldi:2016fql (); Barnard:2016qma (); Komiske:2016rsd (); ATLPHYSPUB2017017 (); Kasieczka:2017nvn (); Bhimji:2017qvb (); Macaluso:2018tck (); Dreyer:2018nbf (); Guest:2016iqz (); Louppe:2017ipp (); Cheng:2017rdo (); Egan:2017ojy (); Fraser:2018ieu (); Almeida:2015jua (); Pearkes:2017hku (); Butter:2017cot (); Roxlo:2018adx () and highlevel Datta:2017rhs (); AguilarSaavedra:2017rzt (); Luo:2017ncs (); Moore:2018lsr (); Datta:2017lxt (); Komiske:2017aww () input observables. Additional tasks include the removal of pileup Komiske:2017ubm (), modelindependent new physics searches Collins:2018epr (); DAgnolo:2018cun (); DeSimone:2018efk (); Hajer:2018kqm (); Farina:2018fyg (); Heimel:2018mkt (), constraining effective field theories Brehmer:2018kdj (); Brehmer:2018eca (), probabilistic and generative modeling of physics processes deOliveira:2017pjk (); Paganini:2017hrr (); deOliveira:2017rwa (); Paganini:2017dwg (); Andreassen:2018apy (), and enhancing existing physics analyses Baldi:2014kfa (); Baldi:2014pta (); Searcy:2015apa (); Santos:2016kno (); Barberio:2017ngd (); Duarte:2018ite (); Lin:2018cin (); Lai:2018ixk (). See Refs. Larkoski:2017jix (); Guest:2018yhq (); Albertsson:2018maf (); Radovic:2018dip (); Sadowski2018 () for more detailed reviews of machine learning in highenergy physics.
Two key choices must be made when using machine learning for a collider task: how to represent the event and how to analyze that representation. These choices are often made together, with examples from collider physics including calorimeter images paired with convolutional neural networks (CNNs) deOliveira:2015xxd (); Baldi:2016fql (); Barnard:2016qma (); Komiske:2016rsd (); ATLPHYSPUB2017017 (); Kasieczka:2017nvn (); Bhimji:2017qvb (); Macaluso:2018tck (); Dreyer:2018nbf (), particle lists paired with recurrent/recursive neural networks (RNNs) Guest:2016iqz (); Louppe:2017ipp (); Cheng:2017rdo (); Egan:2017ojy (); Fraser:2018ieu (), collections of ordered inputs paired with dense neural networks (DNNs) Almeida:2015jua (); Pearkes:2017hku (); Butter:2017cot (); Roxlo:2018adx (); Datta:2017rhs (); AguilarSaavedra:2017rzt (); Luo:2017ncs (); Moore:2018lsr (), and Energy Flow Polynomials (EFPs) paired with linear methods Komiske:2017aww (). One lesson that emerges from this body of work is that any two sufficiently general models, given access to complete enough information, achieve similar performance. In light of this, criteria such as understandability of the model and closeness to theoretical and experimental constructs are of central importance.
Given that events are fundamentally sets of particles, particlelevel inputs such as those used in Refs. Guest:2016iqz (); Louppe:2017ipp (); Cheng:2017rdo (); Egan:2017ojy (); Fraser:2018ieu (); Almeida:2015jua (); Pearkes:2017hku (); Butter:2017cot () are a desirable way of representing an event for use in a model. That said, RNNs and DNNs, the two architectures typically used with particlelevel inputs, each fail to be fully satisfactory methods for processing events: DNNs because they require a fixedsize input and RNNs because they are explicitly dependent on the ordering of the inputs. Though ad hoc workarounds for these problems exist, such as zero padding for DNNs or ordering particles by their transverse momentum () or clustering history for RNNs, an ideal architecture would manifestly respect the permutation symmetry of the problem. Such an architecture would be able to handle variablelength inputs while being inherently symmetric with respect to the ordering of the input particles.
The machine learning community has recently developed (and continues to explore) technology which is ideally suited for representing sets of objects for a model DBLP:conf/acl/IyyerMBD15 (); DBLP:conf/cvpr/QiSMG17 (); DBLP:conf/iccv/RezatofighiGMAD17 (); DBLP:conf/nips/QiYSG17 (); DBLP:conf/nips/ZaheerKRPSS17 (); DBLP:journals/corr/abs170903019 (); DBLP:journals/corr/abs171207262 (); DBLP:conf/aaai/RezatofighiMSD018 (); DBLP:journals/corr/abs180500613 (); DBLP:journals/corr/abs180600050 (). One context where this appears is learning from point clouds, sets of data points in space, which have the same permutation symmetry of collider events and are directly relevant for applications such as selfdriving car technologies. The Deep Sets framework, recently developed in Ref. DBLP:conf/nips/ZaheerKRPSS17 (), demonstrates how permutationinvariant functions of variablelength inputs can be parametrized in a fully general way. The key observation is that the summation function, which is clearly symmetric with respect to the order of the arguments, is general enough to encapsulate all symmetric functions if one is allowed a large enough internal (latent) space.
In the context of a physics observable that is a symmetric function of an arbitrary number of particles each with features, the result from Ref. DBLP:conf/nips/ZaheerKRPSS17 () can be stated as:
Observable Decomposition.
An observable can be approximated arbitrarily well as:
(1) 
where is a perparticle mapping and is a continuous function.
A schematic representation of Eq. (1) is shown in Fig. 1. Inherent in the decomposition of Eq. (1) is a latent space of dimension that serves to embed the particles such that an overall latent event representation is obtained when the sum is carried out. One should think of the features for each particle as possibly being kinematic information, such as the particle’s , rapidity , and azimuthal angle , or other quantum numbers such as the particle’s charge or flavor. Sec. 2 contains additional mathematical details regarding this decomposition.
With a suitable modification of Eq. (1), we can restrict the decomposition to infrared and collinearsafe (IRCsafe) observables:
IRCSafe Observable Decomposition.
An IRCsafe observable can be approximated arbitrarily well as:
(2) 
where is the energy (or ) and the angular information of particle .
The energyweighting factors as well as the energyindependent in Eq. (2) ensure that the event representation in the latent space is IRCsafe.
In this paper, we show that many common observables are naturally encompassed by simple choices of and from Eqs. (1) and (2). Furthermore, we can parametrize and by neural network layers, capable of learning essentially any function, in order to explore more complicated observables. In keeping with the naming convention of Ref. Komiske:2017aww () for methods involving IRCsafe observables, we term a network architecture implementing Eq. (2) an Energy Flow Network (EFN). By contrast, we refer to the more general case of an architecture that implements Eq. (1) as a Particle Flow Network (PFN). Our framework manifestly respects the variable length and permutation invariance of particle sets, achieves performance competitive with existing techniques on key collider tasks, and provides a platform for visualizing the information learned by the model. Beyond this, we demonstrate how our framework unifies the existing event representations of calorimeter images and radiation moments, and we showcase the extraction of novel analytic observables from the trained model.
One everpresent collider phenomenon that involves complicated multiparticle final states is the formation and observation of jets, sprays of colorneutral hadrons resulting from the fragmentation of highenergy quarks and gluons in quantum chromodynamics (QCD). Numerous individual observables have been proposed to study jets including the jet mass, constituent multiplicity, image activity Pumplin:1991kc (), subjettiness Thaler:2010tr (); Thaler:2011gf (), trackbased observables Krohn:2012fg (); Chang:2013rca (), generalized angularities Larkoski:2014pca (), (generalized) energy correlation functions Larkoski:2013eya (); Moult:2016cvt (), soft drop multiplicity Larkoski:2014wba (); Frye:2017yrw (), and many more (see Refs. Abdesselam:2010pt (); Altheimer:2012mn (); Altheimer:2013yza (); Adams:2015hiv (); Larkoski:2017jix (); Asquith:2018igt () for reviews). Machine learning methods have found tremendous applicability to jet classification tasks, greatly outperforming individual standard observables. Jet classification provides an ideal case study for the Deep Sets method in a collider setting since jets, like events, are fundamentally variably sized and invariant under reorderings of their constituents.
To showcase the efficacy of EFNs and PFNs, we apply them to the task of distinguishing lightquark jets from gluon jets Gallicchio:2011xq (); Gallicchio:2012ez (); Aad:2014gea (); Gras:2017jty (), finding that they achieve excellent classification performance. In general, the PFN model outperforms the EFN model, indicating that IRCunsafe information is helpful for discriminating quark and gluon jets. Additionally, including particle identification information improves the classification performance of the PFN. It would be interesting to apply all of these methods in a fullydata driven way Metodiev:2017vrx (); Komiske:2018oaa (); Komiske:2018vkc () to test these conclusions beyond the limited scope of parton shower generators.
One fascinating aspect of EFNs is that they enable a natural visualization of the learned latent space, providing insights as to what exactly the machine is learning. In particular, since the function of an EFN typically takes the twodimensional angular information of a particle as input, this twodimensional space is easily visualized. In the context of quark/gluon discrimination, we observe that the EFN learns a latent representation that “pixelates” the rapidityazimuth plane, dynamically sizing the pixels to be smaller near the core of the jet and larger farther out. We also find qualitative and quantitative evidence that the EFN has in a sense “understood” the collinear singularity structure of QCD.
The rest of this paper is organized as follows. Sec. 2 provides a detailed mathematical discussion of the observable decompositions and explores Eqs. (1) and (2) in the context of specific observables and event representations. Sec. 3 discusses the implementation details of our EFN and PFN architectures, with other models discussed in App. A. Sec. 4 contains the case study discriminating quark and gluoninitiated jets and demonstrates our new techniques for visualizing and analyzing the learned information. Conclusions are presented in Sec. 5. A supplementary top jet tagging study is presented in App. B, and additional visualizations of the models are provided in App. C. The EFN and PFN architectures are available online as part of our EnergyFlow package energyflow () along with example code.
2 A general framework for observables
Events consist of variable numbers of particles with no intrinsic ordering, so observables are described mathematically as functions of sets of particles. Such a mathematical formulation allows for a direct exploration of the space of observables. For instance, Ref. Komiske:2017aww () exploited IRC safety to construct a linear approximating basis of all IRCsafe observables. Here, we treat the entire space of observables (both with and without IRC safety), using their mathematical structure to arrive at a general decomposition relevant for theoretically categorizing observables as well as developing machine learning techniques.
2.1 Observables as functions of sets of particles
The key mathematical fact that we exploit, due to Ref. DBLP:conf/nips/ZaheerKRPSS17 (), is that a generic function of a set of objects can be decomposed to arbitrarily good approximation in a practical and intuitive way. We state this result explicitly below:
Deep Sets Theorem DBLP:conf/nips/ZaheerKRPSS17 ().
Let be compact, be the space of sets with bounded cardinality of elements in , and be a bounded interval. Consider a continuous function that is invariant under permutations of its inputs, i.e. for all and . Then there exists a sufficiently large integer and continuous functions , such that the following holds to an arbitrarily good approximation:^{1}^{1}1It is formally necessary to restrict the domains and ranges of the functions to be compact because the proof of the Deep Sets Theorem, given fully in Ref. DBLP:conf/nips/ZaheerKRPSS17 (), makes use of the StoneWeierstrass polynomial approximation theorem Stone:1948gen (), which applies for compact spaces. After the expansion in polynomials of the features, the result follows by careful application of the fundamental theorem of symmetric polynomials.
(3) 
We only rely on the Deep Sets Theorem to justify the generality of Eq. (3), which can otherwise be regarded as an interesting, manifestly permutationinvariant parameterization.
The Deep Sets Theorem can be immediately applied to the collider physics context where observables are viewed as functions of sets of particles. We denote an event with particles as , where contains the relevant attributes of particle (momentum, charge, flavor, etc.). Phrased in the collider physics language, it states that an observable can be approximated arbitrarily well as:
(4) 
where is a perparticle mapping and is a continuous function. This provides a mathematical justification for the Observable Decomposition stated in Eq. (1).
The content of the Observable Decomposition is that any observable can be viewed as linearly summing over the particles in some internal space and then mapping the result to an output space. We refer to as the latent space and each component of the perparticle mapping as a filter. The latent space could be, for example, the pixel values of a detector image or a moment decomposition of the radiation pattern. Summing over the particles induces a latent description of the entire event, which is mapped by the function to the value of the observable.
Many existing collider observables ranging from event shapes to jet substructure observables naturally fit into the decomposition of Eq. (4). In particular, observables that are defined directly in terms of the particles themselves (i.e. not algorithmically) can often be exactly encompassed. Several examples of such observables are summarized in Table 1. The fact that an exact decomposition holds in these cases indicates that the Observable Decomposition indeed captures an essential aspect of particlelevel collider observables.
Observable  Map  Function  

Mass  
Multiplicity  
Track Mass  
Track Multiplicity  
Jet Charge Krohn:2012fg ()  
Eventropy Larkoski:2014pca ()  
Momentum Dispersion CMS:2013kfa ()  
parameter Parisi:1978eg () 
2.2 Enforcing infrared and collinear safety
We can formulate the Observable Decomposition specifically for a class of observables of particular theoretical interest, namely IRCsafe observables Kinoshita:1962ur (); Lee:1964is (); sterman1995handbook (); Weinberg:1995mt (). IRC safety corresponds to robustness of the observable under collinear splittings of a particle or additions of soft particles, which makes the observable tractable in perturbative quantum field theory as well as robust to experimental resolution effects.
Remarkably, building IRC safety into the latent representation simply corresponds to energyweighting the contributions of each particle and restricting to only depend on the particle geometry . The energyweighting and geometry for particle depends on the collider context. At an collider, it is natural to take and , where is the energy and the fourmomentum. At a hadron collider, it is natural to take and , where is the transverse momentum, is the rapidity, and the azimuthal angle.^{2}^{2}2As discussed in Ref. Komiske:2017aww (), another sensible choice for the angular measure is . Particle mass information, if present, can be passed to a PFN via flavor information. In practice, we typically focus on dimensionless observables and use the appropriate normalized weights: or .
Any IRCsafe observable can be approximated arbitrarily well by the decomposition:
(5) 
where is a perparticle angular mapping and is continuous. All observables of the form in Eq. (5) are manifestly IRC safe due to the energyweighted linear sum structure, the dependence of on purely geometric inputs , and the fact that continuous functions of IRCsafe observables are IRC safe.^{3}^{3}3Ratios of IRCsafe observables are not necessarily IRC safe Larkoski:2013paa (); Larkoski:2015lea () since division is discontinuous at zero.
The fact that the energyweighted decomposition in Eq. (5) suffices to approximate all IRCsafe observables is intuitive from the fact that a continuous function of a sufficiently highresolution calorimeter image can be used to approximate an IRCsafe observable arbitrarily well Tkachov:1995kk (); Sveshnikov:1995vi (); Cherzor:1997ak (). As discussed in Sec. 2.3, an image of the calorimeter deposits is exactly encompassed by the energyweighted observable decomposition.
Here, we provide a direct argument to arrive at Eq. (5), building off the Deep Sets Theorem and following similar logic as Ref. Komiske:2017aww (). Given the decomposition of an IRCsafe observable into and via Eq. (4), the IRC safety of the observable corresponds to the following statements:
(6)  
(7) 
where Eq. (6) holds for all directions that a soft particle could be emitted and Eq. (7) holds for all energy fractions of the collinear splitting. In Eq. (7), we have selected particle 1 to undergo the collinear splitting but the statement holds for any of the particles by permutation symmetry. The equations here only hold to a specified accuracy of approximation in the Observable Decomposition, which we leave implicit since it does not alter the structure of our argument.
We now make the following suggestive redefinition of to ensure that the latent representation of a particle vanishes if the particle has zero energy:
(8) 
Infrared safety via Eq. (6) ensures that the value of the observable is unchanged under this redefinition, so without loss of generality we may take to vanish on arbitrarily soft particles.
Making another convenient redefinition of , we choose a and let:
(9) 
Collinear safety via Eq. (7) ensures that the value of the observable is unchanged under such a redefinition, which holds for any .
We now show that the freedom to redefine the mapping using Eqs. (8) and (9) for an IRCsafe observable leads to the IRCsafe Observable Decomposition in Eq. (5). To see this, consider approximating in the energy argument via the StoneWeierstrass theorem. Calling the angular coefficients of each term yields:
(10) 
for some large but finite . How large must be depends on the specified precision that we have been leaving implicit.
Invoking the soft redefinition in Eq. (8), may be taken to vanish on arbitrarily soft particles, which allows to be set to zero without changing the value of the observable. Implementing the collinear redefinition in Eq. (9) after the expansion in energy, we obtain:
(11) 
From this equation, we seek to argue that for may be taken to vanish. For , this redefinition decreases the higherorder coefficients by a factor of without changing the corresponding observable. Iterated application of this fact allows the higherorder coefficients to be removed while keeping the term linear in the energy. Thus, to an arbitrarily good approximation, we can take for some angular function , which we subsequently rename to .
To summarize, the Deep Sets Theorem, combined with IRC safety, shows that the map can be taken to be linear in energy without loss of generality. Collinear safety was critical in arguing that could be taken to be affine linear in the energy and infrared safety was critical in arguing that the constant piece could be set to zero without loss of generality. This is exactly the result needed to justify the IRCsafe Observable Decomposition in Eq. (5), thereby completing the argument. Beyond potential applications for building IRC safety directly into models, such an observable decomposition for IRCsafe observables may be useful for shedding light on the structure of these important observables more broadly.
2.3 Encompassing image and moment representations
Beyond the single observables tabulated in Table 1, entire event representations can be encompassed in the Observable Decomposition framework as well.
One common event representation is to view events as images by treating the energy deposits in a calorimeter as pixel intensities Cogan:2014oua (); deOliveira:2015xxd (); Baldi:2016fql (); Barnard:2016qma (); Komiske:2016rsd (); ATLPHYSPUB2017017 (). Since typical pixelizations for jet classification are , the images are quite sparse, with an order of magnitude more pixels than particles. Treating the detector as a camera and events as images allows for modern image recognition technology to be applied to collider physics problems. These images are typically fed into a convolutional neural network, which is trained to learn a function of the images optimized for a specific task.
The imagebased event representation of a jet as a collection of pixels fits naturally into the Observable Decomposition. The energy (or transverse momentum) deposited in each pixel is simply a sum over the energies of the particles hitting that pixel. Letting be an indicator function of pixel in the rapidityazimuth plane, we have that the intensity of pixel is:
(12) 
Thus, having be an indicator function for the location of the pixel directly allows the latent representation of the IRCsafe Observable Decomposition to be a detector image. We illustrate this in Fig. 2 for the rapidityazimuth plane relevant for a hadron collider. Here, the filters are a collection of localized square bumps evenly spaced throughout the rapidityazimuth plane.
Another way to represent an event or jet is as a collection of moments of its radiation pattern. Moments (or tensors) have been considered for analyzing hadronic energy flow patterns both for and hadron colliders Fox:1978vu (); Donoghue:1979vi (); GurAri:2011vx (). A momentbased representation has yet to be directly exploited for machine learning applications in collider physics, though is closely related to the EFPs Komiske:2017aww ().^{4}^{4}4There is a rich connection between the moments of the event radiation pattern and multiparticle energy correlators, a detailed discussion of which we leave to future work. See footnote 8 of Ref. Komiske:2017aww (). Here we restrict to the collimated case of jets, but a similar discussion holds at the event level. The moments of the radiation pattern in the rapidityazimuth plane are:
(13) 
This can be manifestly decomposed according to the IRCsafe Observable Decomposition by simply making each filter , as illustrated in Fig. 3. Here, the filters are a collection of nonlocalized functions which weight different parts of the event differently.
More generally, we can visualize for learned IRCsafe latent spaces, where the model itself learns its own event representation. In interpreting these visualizations, it is worth keeping in mind that localized filters like Fig. 2 correspond to an imagelike representation, while global filters like Fig. 3 correspond to a momentlike representation. The flexibility of the IRCsafe Observable Decomposition allows for more complicated filters as well. As we will see in Sec. 4.4, visualizing the latent space is extremely useful in understanding the behavior of EFNs. Moreover, similar (albeit higherdimensional) visualizations can be performed in the general PFN case of and have been explored in the point cloud context DBLP:conf/cvpr/QiSMG17 ().
3 Network implementation
In this section, we describe our implementation and adaptation of the Deep Sets decomposition for use in a particle physics context. In light of the quark versus gluon jet case study presented in Sec. 4, we focus here on inputting individual jets to the model, though we emphasize that the method is broadly applicable at the event level.
3.1 Preprocessing
The goal of preprocessing inputs is to assist the model in its effort to solve an optimization problem. Typically, preprocessing steps are optional, but are applied in order to improve the numerical convergence of the model, given the practical limitations of finite dataset and model size, as well as the particular choice of parameter initialization. The preprocessing described in this section was found to be helpful, and sometimes necessary, for achieving a welltrained EFN or PFN model for the applications considered in Sec. 4. It is likely that for further applications of EFNs or PFNs, such as eventlevel studies, the appropriate preprocessing steps may change.
For the models we construct, kinematic informationtransverse momentum , rapidity , and azimuthal angle are always given for each particle. We preprocess these features as follows: the transverse momenta are divided by the total scalar sum and the rapidities and azimuthal angles are centered based on the rapidity and azimuthal angle of the jet, using the scheme jet axis. In terms of the fourmomentum of each particle, this preprocessing step can be cast into the following suggestive form:
(14) 
with , where the subscripts indicate the rapidity and azimuth of the jet fourvector. This notation makes clear that the perparticle preprocessing of Eq. (14) solely relies on the scalar sum , rapidity, and azimuth of the jet, which itself can be written in terms of an IRCsafe Observable Decomposition with . Alternative jet centerings, such as those based on the weighted centroid, also fit nicely into this framework.^{5}^{5}5These observations motivate an iterative localglobal architecture which learns an event representation, applies it perparticle, and repeats. Such an architecture could explicitly or learnably fold in this preprocessing as a first step. We leave further developments in this direction to future work.
Optionally, the inputs may also include particle identification (ID) information. Though typically encoded using the Particle Data Group (PDG) particle numbering scheme Tanabashi:2018oca (), the large and irregular integer values employed therein are not ideal inputs to a model expecting inputs roughly in the numerical range . Therefore, a mapping from PDG IDs to small floats is performed for each particle (the details of which are provided below). While this approach, which only uses a single feature to encode the particle ID information, should be sufficient to input this information to the model, alternative approaches using multiple categorical features may be easier for the model to interpret, since particle ID is inherently discrete rather than continuous. For instance, using two additional features per particle, one feature could indicate the charge of the particle and the other one could indicate (where corresponds to a hadron, one of ), covering an experimentally realistic particle ID scheme. Onehot encoding of the particle ID is another option.
In order to explore how particle identification is helpful to a model, we use it in four different ways, each with a PFN architecture. We describe each of the different models and levels of information used throughout Sec. 4 below:

PFNID: PFN, adding in the full particle ID information. For the case study in Sec. 4, particles are indicated as being one of , , , , , , , , , , , , , , which are represented to the model as a single float value starting at 0 and increasing by 0.1 for each distinct type, respectively.^{6}^{6}6Note that is absent since we include its decay, usually into two photons.

PFNEx: PFN, adding in experimentally realistic particle ID information. For the case study in Sec. 4, particles are indicated as being one of , , , , , , , , which are represented to the model analogously to the PFNID case.^{7}^{7}7These categories are based on particle flow reconstruction algorithms at ATLAS and CMS Beaudette:2014cea (); Sirunyan:2017ulk (); Aaboud:2017aca (), where and . Additional experimental information, such as separation, feasible at ALICE and LHCb (or at ATLAS and CMS at low ), can carry added information, as could exclusive hadron reconstruction. Particle ID information is typically captured in likelihood ratios for different particle hypotheses, which fits naturally into a categorical encoding scheme where there is a feature for each particletype likelihood ratio.

PFNCh: PFN, adding in the electric charge of the particles as an additional feature.

PFN: The particle flow network using only threemomentum information via Eq. (1).

EFN: The energy flow network using only IRCsafe latent space information via Eq. (2).
3.2 Network architecture
So far, there has not yet been any machine learning in our effort to apply the decompositions in Eqs. (1) and (2) to collider data. The machine learning enters by choosing to approximate the functions and with neural networks.^{8}^{8}8Ref. DBLP:conf/nips/ZaheerKRPSS17 () describes two types of architectures in the Deep Sets framework, termed invariant and equivariant. Equivariance corresponds to producing perparticle outputs that respect permutation symmetry. For this paper, our interest is in the invariant case, but we leave for future work an exploration of the potential particle physics applications of an equivariant architecture. Neural networks are a natural choice to use because sufficiently large neural networks can approximate any wellbehaved function.
To parametrize the functions and in a sufficiently general way, we use several dense neural network layers as universal approximators, as shown in Fig. 4. For , we employ three dense layers with 100, 100, and nodes, respectively, where is the latent dimension that will be varied in powers of 2 up to 256. For , we use three dense layers, each with 100 nodes. We confirmed that several network architectures with more or fewer layers and nodes achieved similar performance. Each dense layer uses the ReLU activation function relu () and Heuniform parameter initialization heuniform (). A twounit layer with a softmax activation function is used as the output layer of the classifier. See App. A for additional details regarding the implementations of the EFN, PFN, and other networks. The EnergyFlow Python package energyflow () contains implementations and examples of EFN and PFN architectures.
4 Discriminating quark and gluon jets
To demonstrate the EFN architecture in a realistic setting, we implement and train an EFN and several PFN variants to discriminate lightquark from gluon initiated jets Gallicchio:2011xq (); Gallicchio:2012ez (); Aad:2014gea (); Gras:2017jty (), a problem relevant for new physics searches as well as precision measurements. See App. B for a similar study on classifying top jets from QCD jets using samples based on Ref. Butter:2017cot ().
4.1 Event generation
The samples used for this study were and events generated with Pythia 8.226 Sjostrand:2006za (); Sjostrand:2014zea () at TeV using the WeakBosonAndParton:qqbar2gmZg and WeakBosonAndParton:qg2gmZq processes, ignoring the photon contribution and requiring the to decay invisibly to neutrinos. Hadronization and multiple parton interactions (i.e. underlying event) were turned on and the default tunings and shower parameters were used. Final state nonneutrino particles were clustered into anti jets Cacciari:2008gp () using FastJet 3.3.0 Cacciari:2011ma (). Jets with GeV and were kept. No detector simulation was performed.^{9}^{9}9In the context of experimental applications, it is worth noting that the different resolutions of different particle types can be naturally accomodated in our framework. While labeling these jets using quark/gluon parton labels is manifestly unphysical, applications of these techniques at colliders could rely on an operational jet flavor definition Komiske:2018vkc () and weak supervision techniques for training directly on data Metodiev:2017vrx (); Komiske:2018oaa () (see also Refs. Dery:2017fap (); Cohen:2017exh (); blanchard2016classification (); blanchard2018corrigendum (); Metodiev:2018ftz ()).
4.2 Classification performance
A standard tool to analyze a classifier is the receiver operating characteristic (ROC) curve, obtained from the true positive and false positive rates as the decision threshold is varied. This may also be plotted as a Significance Improvement (SI) curve Gallicchio:2012ez (), namely as a function of . To condense the performance of a classifier into a single quantity, the area under the ROC curve (AUC) is commonly used, which is also the probability that the classifier correctly sorts randomly drawn signal (quark jet) and background (gluon jet) samples. An AUC of 0.5 corresponds to a random classifier and an AUC of 1.0 corresponds to a perfect classifier. We also report the background rejection at 50% signal efficiency ( at ) as an alternative performance metric.
For each of the models, we sweep the latent dimension of the internal representation from 2 to 256 in powers of 2. As discussed in Sec. 3.1, four PFN models were trained each with different particletype information. Models are trained ten times each to give a sense of the variation and stability of the training. The resulting model performances as quantified by the AUC are shown in Fig. 5. As anticipated, the performance of each model increases as the latent dimension increases, with good performance achieved by . The higher variance at low latent dimensions arises because some of the filters fail to train to nonzero values in those cases. The performance of the models appears to saturate by the larger latent dimensions, which justifies our use of as our benchmark latent dimension size for additional explorations.
In Fig. 6, we show the full ROC and SI curves of these models with latent dimension 256. The best model performance of all tested techniques and models was the PFNID with full particle ID, followed closely by the PFNEx with experimentally realistic particle ID. Figs. 5 and 6 show a welldefined hierarchy of model performances at all latent dimension sizes based on the information available to the model. The fact that the PFNs outperform the EFN indicates that IRCunsafe information is helpful for this discrimination task, which is not surprising in light of the fact that the constituent multiplicity is known to be a good quark/gluon discriminant Gallicchio:2012ez ().
4.3 Comparison to other architectures
Symbol  Name  Short Description 

PFNID  Particle Flow Network w. ID  PFN with full particle ID 
PFNEx  Particle Flow Network w. PF ID  PFN with realistic particle ID 
PFNCh  Particle Flow Network w. charge  PFN with charge information 
PFN  Particle Flow Network  Using threemomentum information 
EFN  Energy Flow Network  Using IRCsafe information 
RNNID  Recurrent Neural Network w. ID  RNN with full particle ID 
RNN  Recurrent Neural Network  Using threemomentum information 
EFP  Energy Flow Polynomials  A linear basis for IRCsafe information 
DNN  Dense Neural Network  Trained on an subjettiness basis 
CNN  Convolutional Neural Network  Trained on grayscale jet images 
Constituent Multiplicity  Number of particles in the jet  
Soft Drop Multiplicity  Probes number of perturbative emissions  
Jet Mass  Mass of the jet 
Model  AUC  at 

PFNID  
PFNEx  
PFNCh  
PFN  
EFN  
RNNID  0.9010  34.4 
RNN  0.8899  30.5 
EFP  0.8919  29.7 
DNN  0.8849  26.4 
CNN  0.8781  25.5 
0.8401  19.0  
0.8297  14.2  
0.7401  7.2 
Besides comparing the EFN and PFN architectures to each other, we also compare the models to a variety of other classifiers, summarized in Table 2 and described in more detail in App. A.
Of particular interest are the RNNID and RNN models, which also take particles as input (with and without full particle ID, respectively), but process them in a way which is dependent on the order the particles were fed into the network (decreasing ordering was used). In Fig. 7, ROC and SI curves are shown for the RNNID and RNN architectures, as well as their natural counterparts, PFNID and PFN. We see that PFNID slightly outperforms RNNID whereas the PFN and RNN are comparable, though we emphasize that making broad conclusions based on this one result is difficult given the variety of different RNN architectures we could have chosen. Since PFNs are less expressive than RNNs, which can learn orderdependent functions, it is satisfying that both the PFN and RNN architectures achieve comparable classification performance with similar information.
The other machine learning architectures we compare to are a DNN trained on the subjettiness basis Thaler:2010tr (); Thaler:2011gf (); Datta:2017rhs (), a CNN trained on jet images Cogan:2014oua (); deOliveira:2015xxd (); Komiske:2016rsd (), and a linear classifier trained on the energy flow basis Komiske:2017aww (). Their performance, as given by their AUC and background rejection at 50% signal efficiency, is summarized in Table 3. Classification improves with the addition of IRCunsafe information, as seen in the gain that the various PFN and RNN models have over the EFN and EFP models. There is also a boost in performance from providing the model with evermore specific particletype information.
Fig. 8 shows ROC and SI curves for all of these models, as well as some common jet substructure observables. The best model is PFNID, followed by RNNID, and then (as shown in this figure) linear EFPs, which, somewhat remarkably, is the best architecture by AUC and SI curve height that does not take particles as direct inputs. We note that for the CNN, one can in principle include particle ID information via additional channels, though training a 14channel CNN is computationally challenging as each channel comes with additional numbers, most of which are zero. Similar to RNNs and CNNs, the EFN and PFN architectures endeavor to be efficient by reducing the number of trainable parameters using weight sharing by applying the same network to each particle. Adding particletype information to the EFPs or the subjettiness DNNs might be possible through a suitable generalization of jet charge Krohn:2012fg (), though we know of no concrete implementation of this in the literature. The fact that PFNs naturally incorporate particle ID information is a important aspect of this architecture.
4.4 Visualizing the singularity structure of QCD
Beyond their excellent classification performance, the EFN and PFN architectures have the additional benefit that the learned function can be directly explored. As discussed in Sec. 2.3, this is particularly true of the EFNs, where is a twodimensional function of the angular information and thus can be directly visualized in the rapidityazimuth plane.
We take the learned network from the best EFN model, as determined by the AUC, and evaluate it at many rapidityazimuth points in the range to form a set of filters representing the learned latent space. We show several of these filters from the EFN models in Fig. 9. These can be directly compared with the corresponding filters for the detector image representation in Fig. 2 and for the radiation moment representation in Fig. 3. Like the image representation, we see that the learned filters are localized bumps in the rapidityazimuth plane, and thus we say that the model appears to have learned a ‘‘pixelization’’ of the rapidityazimuth plane.^{10}^{10}10Note that the ReLU activation function that we used in the model forces the filter values to be positive and allows the model to easily turn off regions of the inputs. Different activation functions may result in different learned latent representations. Unlike the image representation, the ‘‘pixels’’ learned by the model are smaller near the core of the jet and larger near the edge of the jet.
Beyond showing individual filters, it is informative to attempt to visualize an entire EFN latent space at once. This strategy is illustrated in Fig. 10, where each filter is contoured around its 50% value and the contours are overlaid. With the contours plotted simultaneously, the filters learned by the model can all be shown on the same figure, allowing for a direct visualization of the latent space representation.
In Fig. 11, we show this visualization for EFN models with latent dimension varying from 8 to 256 in powers of 2. Some of the filters are zero in the region of interest, perhaps as a result of dying ReLUs, so these are not shown. It is evident from the simultaneous overlay of the filters that their sizes are correlated with their distance from the origin, which is especially clear for the larger latent dimensions. As quark and gluon jets are (approximately) fractal objects with radiation singularly enhanced near the core of the jet as a result of the collinear singularity of QCD, the dynamicallysized pixelization learned by the EFN suggests that the model, in a sense, has understood this fact and adjusted itself accordingly.
To quantify the tendency of the filters to change size as they approach the center of the jet, we plot the area of each filter as a function of its distance from the origin. To define the area of a filter, , we integrate the region of the rapidityazimuth for which the filter is greater than or equal to half of its maximum. To capture a notion of distance from the origin, , we take the distance from the origin to the maximum value of the filter. We exclude filters that have centers outside of the jet radius. The resulting plots of the filters in this space are shown in Fig. 12 for the models with latent space dimension from 8 to 256 in powers of 2. There is a clear linear relationship between the (log) pixel size and the (log) distance to the jet core. In particular, the slope between and is around 1.6 in the cases studied.
We can attempt to understand why the slopes in Fig. 12 are around 2 by considering a uniform pixelization in , where is the distance from the jet axis and is the azimuthal angle around the jet axis (not to be confused with ). As discussed in Ref. Dreyer:2018nbf (), this is the natural emission space of the jet. Translating an area element from this natural emission space to the rapidityazimuth plane yields:
(15) 
Thus, a uniform pixelization in yields the following relationship between the area element (or pixel) size in the rapidityazimuth plane and its distance from the origin:
(16) 
explaining the slopes around 2 observed empirically in Fig. 12. This emergent behavior suggests an interesting connection with recent work on machine learning directly in the emission space of the jet Dreyer:2018nbf (). Deviations from the scaleinvariant expectation of 2 are largest near the core of the jet, where nonperturbative physics or axisrecoil effects Larkoski:2014uqa () become important. The emission plane is visualized directly in App. C, where the pixelization is indeed seen to be highly uniform and regular in that space.
4.5 Extracting new observables from the model
Given that we are able to interpret aspects of the trained EFN in the latent space, we can attempt to go even further and distill the learned filters into closedform observables. Obtaining novel jet substructure observables from machine learning methods has been approached previously by parameterizing an observable and learning the optimal parameters for a particular task Datta:2017lxt (). Here, we go in a different direction and look directly at the latent observables learned by an EFN model.
To make this tractable, we focus on the simple case of a twodimensional latent space. A trained EFN has two learned filters, and , and a learned function . The filters can be visualized in the rapidityazimuth plane and the function can be viewed in the phase space. By studying these visualizations and noting their emergent properties, we can construct observables that reproduce the behavior and predictive power of the trained EFN.
In Figs. (a)a and (b)b, we show the learned filters and of a trained EFN. It is evident that the filters exhibit approximate radial symmetry, with one of the filters concentrated at the center of the jet and the other activated at larger angular distances. Thus, we can restrict our attention to functional forms which depend only on the rapidityazimuth distance from the origin. In particular, due to its builtin IRCsafety, the EFN model has learned filters that correspond to observables of the following approximate form:
(17) 
These are of the general form of IRCsafe angularities Larkoski:2014pca () with a generic radiallysymmetric angular weighting function Gallicchio:2012ez ().^{11}^{11}11Following Ref. Moult:2016cvt (), this section could alternatively be titled “New angles on IRCsafe angularities.” To quantify the filters further, in Fig. (c)c we plot the value of the learned filters as a function of the radial distance, taking an envelope over several radial slices. The complementary central and wideangle nature of the two filters are clearly evident.
By observing the properties of the curves in Fig. (c)c, we fit two IRCsafe observables to the learned profiles of the following forms:
(18) 
with values of , , and found to be suitable. The observables in Eq. (18) are then multiplied by overall factors of 0.60 and 0.18, respectively, to match the arbitrary normalization of the learned filters. While the precise values and shapes of the observable profiles changed from training to training, these general forms emerged for several of the bestperforming models.
The observables and in Eq. (18) are IRCsafe angularities with a linear energy dependence and interesting angular weighting functions. probes the collinear radiation near the core of the jet at angles , and probes wideangle radiation away from the core of the jet at angles . The separate treatment of collinear and wideangle particles is unlike the behavior of the traditional angularities, which have explicit contributions from both collinear and wideangle regions of phase space. Though, as will be shown, each is individually a comparable quark/gluon jet classifier to the traditional angularities, the model is able to combine them in such a way as to achieve a significantly better performance. It would be interesting to perform a firstprinciples QCD study to understand in what sense the separation of collinear and wideangle behavior is beneficial for discrimination.
We now proceed to obtain a closedform estimate of the learned function . In Fig. (a)a, we populate the phase space by quark and gluon jets and color most of the populated space (with a mild threshold against outliers) according to the value of at that phase space point. We also indicate the truth labels of one hundred quark and gluon jets to guide the eye. Based on the transition from red to blue, we can see that the model selects a curved slice through the populated phase space region to obtain its predictions.^{12}^{12}12A similar strategy of fitting analytic functions to learned decision boundaries was carried out in Ref. Likhomanenko:2015aba (). Correspondingly, we choose a closedform estimate based on the simple (squared) Euclidean distance in phase space from a reference point , namely:
(19) 
with and , where, for ease of visualization, a sigmoid is also applied to to monotonically rescale the predictions between 0 and 1 and approximate the value of the learned function. We visualize in Fig. (b)b, finding a satisfying correspondence between the closedform observables and the learned case.
Finally, we study the classification performance of these closedform observables, compared to the trained EFN model and several IRCsafe angularities. We consider three IRCsafe angularities with , , and , which are the jet mass, jet width, and Les Houches angularity, respectively Gras:2017jty (). ROC curves for the various observables are shown in Fig. 15. As single observables, the closedform observables and of Eq. (18) perform similarly to the individual angularities. When combined via of Eq. (19), they outperform all of the single angularities and approach the trained EFN performance. By contrast, when the three angularities are combined with a Boosted Decision Tree (BDT), there is not a significant improvement in the classification performance.^{13}^{13}13The BDT classifier was implemented using scikitlearn scikitlearn () using 100 estimators, trained and tested on 100,000 quark and gluon jets with a 5050 train/test split. We also tested combining and with a BDT, which resulted in similar performance to the closedform and learned cases, verifying that the information is being adequately captured by . The fact that and can be combined to significantly increase the discrimination power indicates that, unlike the considered angularities, the new closedform observables probe complementary information.
5 Conclusions
In this paper, we presented a new technique to learn from collider events in their natural representation as sets of particles. Our approach relies on the Deep Sets theorem DBLP:conf/nips/ZaheerKRPSS17 (), which guarantees that a generic symmetric function can be represented by an additive latent space. In the context of particlelevel collider observables, each particle is mapped to a latent representation and then summed over, and observables are then functions on that latent space. This decomposition encompassed a wide variety of existing event and jetlevel collider observables and representations, including imagebased and momentbased methods.
While these observable decompositions are interesting in their own right, parameterizing them with neural networks yields a machine learning framework ideally suited for learning from variablelength unordered lists of particles. We proposed two fundamental network variants. The IRCsafe EFNs treat each latent space observable as an energyweighted function of geometry, ensuring IRC safety in the latent space by construction. The fullygeneral PFNs are able to incorporate additional particlelevel information such as charge and flavor, maximizing the information available to achieve collider tasks. We showcased the efficacy of these method in a quark/gluon discrimination case study, achieving favorable performance compared with many existing machine learning techniques for particle physics.
A fascinating aspect of the EFNs is that one can directly visualize the perparticle mapping into the latent space. This allowed us to peer inside the network and discover that, in the context of quark/gluon discrimination, it had learned a dynamicallysized pixelization of the rapidityazimuth plane. This pixelization was reminiscent of a jet image but did not adhere to the strict rectilinear grid imposed by traditional jet images. We also found some compelling evidence that the model had ‘‘understood’’ a key property of QCD, specifically its famous collinear singularity structure, since filters closer to the center of the jet were more finely resolved. The relationship between the size and position of the filters could be roughly understood as a uniform pixelization in the natural angular emission plane.
We also presented a simple example of directly learning the physics used by the trained model and obtaining new closedform observables. The IRCsafe Observable Decomposition, combined with the visualizability of the EFN filters, provides a general way to access what the model has learned. In the case of an EFN, the model learned to separately probe collinear and wideangle radiation and then use a phase space distance to classify quarks and gluons. While the quark/gluon classification performance of and is not yet competitive with other simple observables such as the constituent multiplicity, together they notably outperform other IRCsafe angularities. Even though our numerical choices of parameters are specific to these quark/gluon jet samples, it may be interesting to explore the and classes of observables further in the context of theoretical efforts to jointly explore and understand the correlations between two angularities Larkoski:2014pca (); Larkoski:2014tva (); Procura:2018zpn ().
We conclude by discussing possible extensions and additional applications of these methods. Pileup or underlying event mitigation could be an interesting avenue to explore with this architecture, similar to Ref. Komiske:2017ubm () but also to PUPPI Bertolini:2014bba () in that a perparticle correction factor could be applied. Such an effort would need to make use of the equivariant designs of Ref. DBLP:conf/nips/ZaheerKRPSS17 (), which have a separate output for each input. One may also consider adding highlevel information  such as the jet fourmomentum, individual substructure observables, or clustering histories  directly to or in order to improve the network performance. Further, while we used jetlevel examples for our case study, the problem of learning functions of sets applies at the event level more broadly. At the event level, new challenges arise that the simple preprocessing performed here for the jet case studies may not completely solve. An iterative deep sets technique, where multiple latent spaces are constructed, may help to capture the local/global intuition that collider physicists regularly employ to study the hierarchical relationships between events, jets, and particles.
We are grateful to Kyle Cranmer, Frédéric Dreyer, Felice Frankel, Philip Harris, Michael Kagan, Gregor Kasieczka, Markus Klute, Andrew Larkoski, Benjamin Nachman, Siddharth Narayanan, Daniel Roberts, Michael Williams, and Manzil Zaheer for helpful conversations. We are also grateful to Gregor Kasieczka for providing us with the top samples used in App. B. We thank the Harvard Center for the Fundamental Laws of Nature for hospitality while this work was completed. This work was supported by the Office of Nuclear Physics of the U.S. Department of Energy (DOE) under grant DESC0011090 and by the DOE Office of High Energy Physics under grant DESC0012567. JT is supported by the Simons Foundation through a Simons Fellowship in Theoretical Physics. Cloud computing resources were provided through a Microsoft Azure for Research award.
Appendix A Details of models and observables
In this appendix, we describe the details of the machine learning models and observables. All of the models employed for comparison with the EFN/PFN models were applied to our quark/gluon samples in Sec. 4, but only the linear EFP model will be applied to the top samples in App. B. For the substructure observables, the jet mass and constituent multiplicity are easily obtained from the jet in FastJet. Using FastJet contrib 1.033 fjcontrib (), the RecursiveTools 2.0.0beta1 module was used to calculate the soft drop multiplicity Frye:2017yrw () with parameters , , and . The Nsubjettiness 2.2.4 module was used to calculate all subjettiness values Thaler:2010tr (); Thaler:2011gf (). All neural networks were implemented in Keras keras () with the TensorFlow tensorflow () backend and trained on NVIDIA Tesla K80 GPUs on Microsoft Azure.
Here, we provide details for the machine learning models listed in Table 2:

EFN, PFNs: Both the EFN and PFN architectures are contained as part of our EnergyFlow Python package energyflow (), which contains examples demonstrating their training, evaluation, and use. Keras requires contiguous NumPy numpy () arrays as input, so the events are padded with allzero particles to ensure they have equal length. This zeropadding is a technical trick, not a conceptual limitation. It is distinct from the zeropadding necessary to use variablelength inputs in a DNN. This operation can be done on a perbatch level, but to avoid repeating this procedure at each epoch, we zeropad all the events once at the beginning. The Keras Masking layer is used to ignore allzero particles during the training and evaluation of the network. The Keras TimeDistributed layer is used to apply the function to each particle, which in the relevant language is a ‘‘time step’’. To carry out the summation in the latent space, we use the Keras Dot layer for the EFN architecture, to product and sum the transverse momentum fractions with the latent observables, and the Lambda layer for the PFN architecture, to sum over the particles. It may be interesting to consider alternative ‘‘pooling’’ functions to summation, such as maxpooling or averagepooling as in Refs. DBLP:journals/corr/abs170903019 (); DBLP:conf/cvpr/QiSMG17 ().

RNNID, RNN: Recurrent Neural Networks provide an alternative way to process variablelength inputs, albeit one that is explicitly not agnostic to the order of the inputs. We choose to order the particles by their transverse momenta and train two variants: one with only kinematic information (RNN) and one with kinematic as well as full particle ID information (RNNID). The former should be compared with the PFN model and the latter to the PFNID model. We did not consider alternative particle orderings for the RNN, though other investigations have found that performance is robust to choices in particle ordering Louppe:2017ipp (); Andreassen:2018apy (). Our RNN architecture consists of an LSTM layer with 64 nodes (the performance was insensitive to changing this value to 128 or 256) followed by three fullyconnected layers each with 100 nodes and a fullyconnected output layer with 2 nodes. Due to the Masking layer employed, the significantly faster CuDNNLSTM layer was not used and the batch size was taken to be 2000 to help speed up training. An architecture using two or three SimpleRNN layers was also tried but the performance was not as good as with an LSTM layer. The long training time of the RNN models was prohibitive in exploring additional hyperparameters.

EFPs: The energy flow basis Komiske:2017aww () is a linear basis for the space of IRCsafe jet substructure observables, allowing linear methods to be applied to a set of Energy Flow Polynomials with good performance. The EnergyFlow 0.8.2 Python package energyflow () was used to compute all EFPs with degree and complexity , using the normalized default hadronic measure with . The same EFPs with were also tested and found to perform slightly worse. These 996 EFPs, including the trivial constant EFP, were used to train a Fisher’s Linear Discriminant model from the scikitlearn package scikitlearn ().

DNN: The subjettiness basis Datta:2017rhs () is a body phase space basis consisting of the following observables:
(20) We use and, following Ref. Datta:2017rhs (), compute the subjettiness values with respect to axes. A Dense Neural Network consisting of three fullyconnected layers each with 100 nodes and one fullyconnected output layer with 2 nodes was trained on this set of subjettiness values.

CNN: Following Ref. Komiske:2016rsd (), we compute onechannel (grayscale) jet images in a patch of the rapidityazimuth plane. Images were preprocessed as in Refs. Komiske:2016rsd (); Komiske:2018vkc () by subtracting the mean image of the training set and dividing by the perpixel standard deviation. A Convolutional Neural Network architecture was trained on the jet images consisting of three convolutional layers with 48, 32, and 32 filters of size , , and , respectively. These layers were followed by two fullyconnected layers each with 50 nodes and one fullyconnected output layer with 2 nodes. Maxpooling of size and dropout with a rate of 0.1 was implemented after each convolutional layer.
Model  Time per Epoch (sec)  Number of Epochs 

PFNID  73  47 
PFNEx  67  37 
PFNCh  64  51 
PFN  71  34 
EFN  74  54 
RNNID  192  61 
RNN  188  96 
DNN  9  39 
CNN  63  22 
For all quark/gluon models, one million jets were used for training, 200k for validation (except for the EFPs, which did not use validation), and 200k for testing. Common properties to all neural networks include the use of ReLU relu () activation functions for all nonoutput layers, a softmax activation function on the 2 node output layer, Heuniform weight initialization heuniform (), the categorical crossentropy loss function, the Adam optimization algorithm adam (), and a learning rate of 0.001. All nonRNN networks had a batch size of 500; all nonCNN networks had a patience parameter of 8 with the CNN having a patience parameter of 5. Each nonEFN/PFN model was trained twice and the model with the higher AUC on the test set is reported (except when at values are reported, in which case the model with the value of this statistic is reported). In Table 4, we report the typical training times for each of the models.
Appendix B Tagging top jets
In this appendix, we train EFN and PFN models to discriminate top jets from the QCD jet background to provide an additional demonstration of the excellent performance of these models. Since top jets do not have any singularities associated with radiation about their center, training EFN models on tops provides an important cross check about our conclusions in Sec. 4.4 for how the model is learning to pixelize the rapidityazimuth plane.
The top and QCD jets used in this study are based on the dataset used in Ref. Butter:2017cot (), which were provided to us by those authors. The jets are Pythiagenerated, anti, jets at TeV with no underlying event or pileup using a Delphes deFavereau:2013fsa () detector fastsimulation of the ATLAS detector. Jets are kept if they have transverse momentum GeV and pseudorapidity and if they can be matched to a top parton to within a rapidityazimuth distance of . The jets are required to be fully merged, with the three quarks from the top decay contained within the jet radius. See Ref. Butter:2017cot () for additional information about the samples and their generation details, as well as information about other toptagging algorithms. There are 1.2 million training events, with 400k validation events and 400k test events. The jet samples do not contain particlelevel flavor or charge information, and thus we can only train EFN and PFN models that make use of kinematic information alone. A linear EFP model is also trained for comparison.
Given the different topology typical of top jets compared to quark or gluon jets, we implement some additional preprocessing steps designed to help the model train more efficiently. As with the quark and gluon jets of Sec. 4, we center all of the jets in the rapidityazimuth plane based on the fourmomentum of the jet, and we normalize the transverse momenta of the particles to sum to one. Models were trained with just this minimal preprocessing, as well as with additional rotation and reflection (r.r.) operations. For the EFNr.r. and PFNr.r. models, rotations were performed to align the leading eigenvector of the twodimensional moment of the radiation pattern along the vertical axis, and reflections were performed about the horizontal and vertical axes to place the highest quadrant in a consistent quadrant.
The results of training EFN and PFN models for top tagging are shown in Fig. 16 with the latent dimension of the models varying from 8 to 256 in powers of 2. ROC and SI curves of the trained models are shown in Fig. 17 and also compared to the linear EFP model. Performance competitive with the results in Ref. Butter:2017cot () is achieved, particularly for the PFN models. The preprocessing step of rotating and reflecting can be seen to notably improve both the EFN and PFN classification performance. In order to facilitate future comparisons with other models trained on these samples, in Table 5 we report the AUC and gluon background rejection factor at both 0.3 and 0.5 quark efficiency.
Model  AUC  at  at 

PFNr.r.  
PFN  
EFNr.r.  
EFN  
EFPs  0.9803  184  384 
For the EFNs, we can also visualize the learned filters using the technique of Sec. 4.4. The resulting filter visualizations for latent dimension 256 are shown in Fig. 18. The learned filters have some tendency to be smaller near the center and larger near the periphery, but not nearly as much as in the quark/gluon discrimination case in Fig. 10. This is expected because given the typical threeprong topology of a boosted top jet, the jet axis does not have any distinguished radiation associated with it, unlike for a QCD jet where the jet axis tends to lie along a core of radiation. To quantify this effect, we look at the size of the filters as a function of their distance from the origin, shown in Fig. 19. The relationship in the top tagging case is much weaker than the linear relationship present in the quark/gluon discrimination study, with significantly worse linear fits.
Appendix C Additional visualizations
For the quark/gluon EFN model in Sec. 4, the observed relationship between the size and location of the filters qualitatively (in Fig. 11) and quantitatively (in Fig. 12) suggested that the model learned a uniform pixelization in the emission plane. (The top tagging EFN model in App. B did not exhibit as clear of a relationship in either Fig. 18 or Fig. 19.) To directly visualize the EFN filters in the appropriate space, we implement a change of variables from Cartesian coordinates to polar coordinates. The visualizations in Fig. 20 use the same contouring and overlaying technique of Fig. 10 to demonstrate the roughly uniform pixelation in the emission plane for the quark/gluon EFN models with latent dimensions of 16, 64, and 256. In this emission plane, the learned filters can be seen to be much more uniform in size and location compared to the rapidityazimuth plane. The uniformity as a function of indicates that the set of filters approximately has rotational symmetry.
References
 (1) CMS collaboration, F. Beaudette, The CMS Particle Flow Algorithm, in Proceedings, International Conference on Calorimetry for the High Energy Frontier (CHEF 2013): Paris, France, April 2225, 2013, pp. 295304, 2013, 1401.8155.
 (2) CMS collaboration, A. M. Sirunyan et al., Particleflow reconstruction and global event description with the CMS detector, JINST 12 (2017) P10003 [1706.04965].
 (3) ATLAS collaboration, M. Aaboud et al., Jet reconstruction and performance using particle flow with the ATLAS Detector, Eur. Phys. J. C77 (2017) 466 [1703.10485].
 (4) J. Cogan, M. Kagan, E. Strauss and A. Schwarztman, JetImages: Computer Vision Inspired Techniques for Jet Tagging, JHEP 02 (2015) 118 [1407.5675].
 (5) L. de Oliveira, M. Kagan, L. Mackey, B. Nachman and A. Schwartzman, Jetimages  deep learning edition, JHEP 07 (2016) 069 [1511.05190].
 (6) P. Baldi, K. Bauer, C. Eng, P. Sadowski and D. Whiteson, Jet Substructure Classification in HighEnergy Physics with Deep Neural Networks, Phys. Rev. D93 (2016) 094034 [1603.09349].
 (7) J. Barnard, E. N. Dawe, M. J. Dolan and N. Rajcic, Parton Shower Uncertainties in Jet Substructure Analyses with Deep Neural Networks, Phys. Rev. D95 (2017) 014018 [1609.00607].
 (8) P. T. Komiske, E. M. Metodiev and M. D. Schwartz, Deep learning in color: towards automated quark/gluon jet discrimination, JHEP 01 (2017) 110 [1612.01551].
 (9) ATLAS collaboration, Quark versus Gluon Jet Tagging Using Jet Images with the ATLAS Detector, Tech. Rep. ATLPHYSPUB2017017, CERN, Geneva, Jul, 2017.
 (10) G. Kasieczka, T. Plehn, M. Russell and T. Schell, Deeplearning Top Taggers or The End of QCD?, JHEP 05 (2017) 006 [1701.08784].
 (11) W. Bhimji, S. A. Farrell, T. Kurth, M. Paganini, Prabhat and E. Racah, Deep Neural Networks for Physics Analysis on lowlevel wholedetector data at the LHC, in 18th International Workshop on Advanced Computing and Analysis Techniques in Physics Research (ACAT 2017) Seattle, WA, USA, August 2125, 2017, 2017, 1711.03573.
 (12) S. Macaluso and D. Shih, Pulling Out All the Tops with Computer Vision and Deep Learning, 1803.00107.
 (13) F. A. Dreyer, G. P. Salam and G. Soyez, The Lund Jet Plane, 1807.04758.
 (14) D. Guest, J. Collado, P. Baldi, S.C. Hsu, G. Urban and D. Whiteson, Jet Flavor Classification in HighEnergy Physics with Deep Neural Networks, Phys. Rev. D94 (2016) 112002 [1607.08633].
 (15) G. Louppe, K. Cho, C. Becot and K. Cranmer, QCDAware Recursive Neural Networks for Jet Physics, 1702.00748.
 (16) T. Cheng, Recursive Neural Networks in Quark/Gluon Tagging, Comput. Softw. Big Sci. 2 (2018) 3 [1711.02633].
 (17) S. Egan, W. Fedorko, A. Lister, J. Pearkes and C. Gay, Long ShortTerm Memory (LSTM) networks with jet constituents for boosted top tagging at the LHC, 1711.09059.
 (18) K. Fraser and M. D. Schwartz, Jet Charge and Machine Learning, 1803.08066.
 (19) L. G. Almeida, M. Backović, M. Cliche, S. J. Lee and M. Perelstein, Playing Tag with ANN: Boosted Top Identification with Pattern Recognition, JHEP 07 (2015) 086 [1501.05968].
 (20) J. Pearkes, W. Fedorko, A. Lister and C. Gay, Jet Constituents for Deep Neural Network Based Top Quark Tagging, 1704.02124.
 (21) A. Butter, G. Kasieczka, T. Plehn and M. Russell, Deeplearned Top Tagging with a Lorentz Layer, SciPost Phys. 5 (2018) 028 [1707.08966].
 (22) T. Roxlo and M. Reece, Opening the black box of neural nets: case studies in stop/top discrimination, 1804.09278.
 (23) K. Datta and A. Larkoski, How Much Information is in a Jet?, JHEP 06 (2017) 073 [1704.08249].
 (24) J. A. AguilarSaavedra, J. H. Collins and R. K. Mishra, A generic antiQCD jet tagger, JHEP 11 (2017) 163 [1709.01087].
 (25) H. Luo, M.x. Luo, K. Wang, T. Xu and G. Zhu, Quark jet versus gluon jet: deep neural networks with highlevel features, 1712.03634.
 (26) L. Moore, K. Nordström, S. Varma and M. Fairbairn, Reports of My Demise Are Greatly Exaggerated: subjettiness Taggers Take On Jet Images, 1807.04769.
 (27) K. Datta and A. J. Larkoski, Novel Jet Observables from Machine Learning, JHEP 03 (2018) 086 [1710.01305].
 (28) P. T. Komiske, E. M. Metodiev and J. Thaler, Energy flow polynomials: A complete linear basis for jet substructure, JHEP 04 (2018) 013 [1712.07124].
 (29) P. T. Komiske, E. M. Metodiev, B. Nachman and M. D. Schwartz, Pileup Mitigation with Machine Learning (PUMML), JHEP 12 (2017) 051 [1707.08600].
 (30) J. H. Collins, K. Howe and B. Nachman, CWoLa Hunting: Extending the Bump Hunt with Machine Learning, 1805.02664.
 (31) R. T. D’Agnolo and A. Wulzer, Learning New Physics from a Machine, 1806.02350.
 (32) A. De Simone and T. Jacques, Guiding New Physics Searches with Unsupervised Learning, 1807.06038.
 (33) J. Hajer, Y.Y. Li, T. Liu and H. Wang, Novelty Detection Meets Collider Physics, 1807.10261.
 (34) M. Farina, Y. Nakai and D. Shih, Searching for New Physics with Deep Autoencoders, 1808.08992.
 (35) T. Heimel, G. Kasieczka, T. Plehn and J. M. Thompson, QCD or What?, 1808.08979.
 (36) J. Brehmer, K. Cranmer, G. Louppe and J. Pavez, Constraining Effective Field Theories with Machine Learning, Phys. Rev. Lett. 121 (2018) 111801 [1805.00013].
 (37) J. Brehmer, K. Cranmer, G. Louppe and J. Pavez, A Guide to Constraining Effective Field Theories with Machine Learning, Phys. Rev. D98 (2018) 052004 [1805.00020].
 (38) L. de Oliveira, M. Paganini and B. Nachman, Learning Particle Physics by Example: LocationAware Generative Adversarial Networks for Physics Synthesis, Comput. Softw. Big Sci. 1 (2017) 4 [1701.05927].
 (39) M. Paganini, L. de Oliveira and B. Nachman, Accelerating Science with Generative Adversarial Networks: An Application to 3D Particle Showers in Multilayer Calorimeters, Phys. Rev. Lett. 120 (2018) 042003 [1705.02355].
 (40) L. de Oliveira, M. Paganini and B. Nachman, Controlling Physical Attributes in GANAccelerated Simulation of Electromagnetic Calorimeters, in 18th International Workshop on Advanced Computing and Analysis Techniques in Physics Research (ACAT 2017) Seattle, WA, USA, August 2125, 2017, 2017, 1711.08813.
 (41) M. Paganini, L. de Oliveira and B. Nachman, CaloGAN : Simulating 3D high energy particle showers in multilayer electromagnetic calorimeters with generative adversarial networks, Phys. Rev. D97 (2018) 014021 [1712.10321].
 (42) A. Andreassen, I. Feige, C. Frye and M. D. Schwartz, JUNIPR: a Framework for Unsupervised Machine Learning in Particle Physics, 1804.09720.
 (43) P. Baldi, P. Sadowski and D. Whiteson, Searching for Exotic Particles in HighEnergy Physics with Deep Learning, Nature Commun. 5 (2014) 4308 [1402.4735].
 (44) P. Baldi, P. Sadowski and D. Whiteson, Enhanced Higgs Boson to Search with Deep Learning, Phys. Rev. Lett. 114 (2015) 111801 [1410.3469].
 (45) J. Searcy, L. Huang, M.A. Pleier and J. Zhu, Determination of the polarization fractions in using a deep machine learning technique, Phys. Rev. D93 (2016) 094033 [1510.01691].
 (46) R. Santos, M. Nguyen, J. Webster, S. Ryu, J. Adelman, S. Chekanov et al., Machine learning techniques in searches for in the decay channel, JINST 12 (2017) P04014 [1610.03088].
 (47) E. Barberio, B. Le, E. RichterWas, Z. Was, D. Zanzi and J. Zaremba, Deep learning approach to the Higgs boson CP measurement in decay and associated systematics, Phys. Rev. D96 (2017) 073002 [1706.07983].
 (48) J. Duarte et al., Fast inference of deep neural networks in FPGAs for particle physics, JINST 13 (2018) P07027 [1804.06913].
 (49) J. Lin, M. Freytsis, I. Moult and B. Nachman, Boosting with Machine Learning, 1807.10768.
 (50) Y. S. Lai, Automated Discovery of Jet Substructure Analyses, 1810.00835.
 (51) A. J. Larkoski, I. Moult and B. Nachman, Jet Substructure at the Large Hadron Collider: A Review of Recent Advances in Theory and Machine Learning, 1709.04464.
 (52) D. Guest, K. Cranmer and D. Whiteson, Deep Learning and its Application to LHC Physics, 1806.11484.
 (53) K. Albertsson et al., Machine Learning in High Energy Physics Community White Paper, 1807.02876.
 (54) A. Radovic, M. Williams, D. Rousseau, M. Kagan, D. Bonacorsi, A. Himmel et al., Machine learning at the energy and intensity frontiers of particle physics, Nature 560 (2018) 41.
 (55) P. Sadowski and P. Baldi, Deep Learning in the Natural Sciences: Applications to Physics, pp. 269297. Springer International Publishing, Cham, 2018.
 (56) M. Iyyer, V. Manjunatha, J. L. BoydGraber and H. D. III, Deep unordered composition rivals syntactic methods for text classification, in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 2631, 2015, Beijing, China, Volume 1: Long Papers, pp. 16811691, 2015.
 (57) C. R. Qi, H. Su, K. Mo and L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation, in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 2126, 2017, pp. 7785, 2017, DOI.
 (58) S. H. Rezatofighi, V. K. B. G, A. Milan, E. Abbasnejad, A. R. Dick and I. D. Reid, Deepsetnet: Predicting sets with deep neural networks, in IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 2229, 2017, pp. 52575266, 2017, DOI.
 (59) C. R. Qi, L. Yi, H. Su and L. J. Guibas, Pointnet++: Deep hierarchical feature learning on point sets in a metric space, in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 49 December 2017, Long Beach, CA, USA, pp. 51055114, 2017, 1706.02413.
 (60) M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Póczos, R. R. Salakhutdinov and A. J. Smola, Deep sets, in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 49 December 2017, Long Beach, CA, USA, pp. 33943404, 2017, 1703.06114.
 (61) A. Gardner, J. Kanno, C. A. Duncan and R. R. Selmic, Classifying unordered feature sets with convolutional deep averaging networks, 1709.03019.
 (62) Y. Yang, C. Feng, Y. Shen and D. Tian, Foldingnet: Interpretable unsupervised learning on 3d point clouds, 1712.07262.
 (63) S. H. Rezatofighi, A. Milan, Q. Shi, A. R. Dick and I. D. Reid, Joint learning of set cardinality and state distribution, in Proceedings of the ThirtySecond AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 27, 2018, 2018, 1709.04093.
 (64) S. H. Rezatofighi, R. Kaskman, F. T. Motlagh, Q. Shi, D. Cremers, L. LealTaixé et al., Deep permset net: Learn to predict sets with unknown permutation and cardinality using deep neural networks, 1805.00613.
 (65) A. Cotter, M. R. Gupta, H. Jiang, J. Muller, T. Narayan, S. Wang et al., Interpretable set functions, 1806.00050.
 (66) J. Pumplin, How to tell quark jets from gluon jets, Phys. Rev. D44 (1991) 2025.
 (67) J. Thaler and K. Van Tilburg, Identifying Boosted Objects with Nsubjettiness, JHEP 03 (2011) 015 [1011.2268].
 (68) J. Thaler and K. Van Tilburg, Maximizing Boosted Top Identification by Minimizing Nsubjettiness, JHEP 02 (2012) 093 [1108.2701].
 (69) D. Krohn, M. D. Schwartz, T. Lin and W. J. Waalewijn, Jet Charge at the LHC, Phys. Rev. Lett. 110 (2013) 212001 [1209.2421].
 (70) H.M. Chang, M. Procura, J. Thaler and W. J. Waalewijn, Calculating TrackBased Observables for the LHC, Phys. Rev. Lett. 111 (2013) 102002 [1303.6637].
 (71) A. J. Larkoski, J. Thaler and W. J. Waalewijn, Gaining (Mutual) Information about Quark/Gluon Discrimination, JHEP 11 (2014) 129 [1408.3122].
 (72) A. J. Larkoski, G. P. Salam and J. Thaler, Energy Correlation Functions for Jet Substructure, JHEP 06 (2013) 108 [1305.0007].
 (73) I. Moult, L. Necib and J. Thaler, New Angles on Energy Correlation Functions, JHEP 12 (2016) 153 [1609.07483].
 (74) A. J. Larkoski, S. Marzani, G. Soyez and J. Thaler, Soft Drop, JHEP 05 (2014) 146 [1402.2657].
 (75) C. Frye, A. J. Larkoski, J. Thaler and K. Zhou, Casimir Meets Poisson: Improved Quark/Gluon Discrimination with Counting Observables, JHEP 09 (2017) 083 [1704.06266].
 (76) A. Abdesselam et al., Boosted objects: A Probe of beyond the Standard Model physics, Eur. Phys. J. C71 (2011) 1661 [1012.5412].
 (77) A. Altheimer et al., Jet Substructure at the Tevatron and LHC: New results, new tools, new benchmarks, J. Phys. G39 (2012) 063001 [1201.0008].
 (78) A. Altheimer et al., Boosted objects and jet substructure at the LHC. Report of BOOST2012, held at IFIC Valencia, 23rd27th of July 2012, Eur. Phys. J. C74 (2014) 2792 [1311.2708].
 (79) D. Adams et al., Towards an Understanding of the Correlations in Jet Substructure, Eur. Phys. J. C75 (2015) 409 [1504.00679].
 (80) L. Asquith et al., Jet Substructure at the Large Hadron Collider : Experimental Review, 1803.06991.
 (81) J. Gallicchio and M. D. Schwartz, Quark and Gluon Tagging at the LHC, Phys. Rev. Lett. 107 (2011) 172001 [1106.3076].
 (82) J. Gallicchio and M. D. Schwartz, Quark and Gluon Jet Substructure, JHEP 04 (2013) 090 [1211.7038].
 (83) ATLAS collaboration, G. Aad et al., Lightquark and gluon jet discrimination in collisions at with the ATLAS detector, Eur. Phys. J. C74 (2014) 3023 [1405.6583].
 (84) P. Gras, S. Höche, D. Kar, A. Larkoski, L. Lönnblad, S. Plätzer et al., Systematics of quark/gluon tagging, JHEP 07 (2017) 091 [1704.03878].
 (85) E. M. Metodiev, B. Nachman and J. Thaler, Classification without labels: Learning from mixed samples in high energy physics, JHEP 10 (2017) 174 [1708.02949].
 (86) P. T. Komiske, E. M. Metodiev, B. Nachman and M. D. Schwartz, Learning to classify from impure samples with highdimensional data, Phys. Rev. D98 (2018) 011502 [1801.10158].
 (87) P. T. Komiske, E. M. Metodiev and J. Thaler, An operational definition of quark and gluon jets, 1809.01140.
 (88) ‘‘EnergyFlow.’’ https://energyflow.network.
 (89) M. H. Stone, The generalized weierstrass approximation theorem, Mathematics Magazine 21 237.
 (90) CMS collaboration, CMS, Performance of quark/gluon discrimination in 8 TeV pp data, Tech. Rep. CMSPASJME13002, CERN, 2013.
 (91) G. Parisi, Super Inclusive CrossSections, Phys. Lett. 74B (1978) 65.
 (92) T. Kinoshita, Mass singularities of Feynman amplitudes, J. Math. Phys. 3 (1962) 650.
 (93) T. D. Lee and M. Nauenberg, Degenerate Systems and Mass Singularities, Phys. Rev. 133 (1964) B1549.
 (94) G. Sterman, J. Smith, J. C. Collins, J. Whitmore, R. Brock, J. Huston et al., Handbook of perturbative qcd, Reviews of Modern Physics 67 (1995) 157.
 (95) S. Weinberg, The Quantum theory of fields. Vol. 1: Foundations. Cambridge University Press, 2005.
 (96) A. J. Larkoski and J. Thaler, Unsafe but Calculable: Ratios of Angularities in Perturbative QCD, JHEP 09 (2013) 137 [1307.1699].
 (97) A. J. Larkoski, S. Marzani and J. Thaler, Sudakov Safety in Perturbative QCD, Phys. Rev. D91 (2015) 111501 [1502.01719].
 (98) F. V. Tkachov, Measuring multi  jet structure of hadronic energy flow or What is a jet?, Int. J. Mod. Phys. A12 (1997) 5411 [hepph/9601308].
 (99) N. A. Sveshnikov and F. V. Tkachov, Jets and quantum field theory, Phys. Lett. B382 (1996) 403 [hepph/9512370].
 (100) P. S. Cherzor and N. A. Sveshnikov, Jet observables and energy momentum tensor, in Quantum field theory and highenergy physics. Proceedings, Workshop, QFTHEP’97, Samara, Russia, September 410, 1997, pp. 402407, 1997, hepph/9710349.
 (101) G. C. Fox and S. Wolfram, Observables for the Analysis of Event Shapes in e+ e Annihilation and Other Processes, Phys. Rev. Lett. 41 (1978) 1581.
 (102) J. F. Donoghue, F. E. Low and S.Y. Pi, Tensor Analysis of Hadronic Jets in Quantum Chromodynamics, Phys. Rev. D20 (1979) 2759.
 (103) G. GurAri, M. Papucci and G. Perez, Classification of Energy Flow Observables in Narrow Jets, 1101.2905.
 (104) ParticleDataGroup collaboration, M. Tanabashi et al., Review of Particle Physics, Phys. Rev. D98 (2018) 030001.
 (105) V. Nair and G. E. Hinton, Rectified linear units improve restricted boltzmann machines, in Proceedings of the 27th international conference on machine learning (ICML10), pp. 807814, 2010.
 (106) K. He, X. Zhang, S. Ren and J. Sun, Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification, in Proceedings of the IEEE international conference on computer vision, pp. 10261034, 2015.
 (107) T. Sjostrand, S. Mrenna and P. Z. Skands, PYTHIA 6.4 Physics and Manual, JHEP 05 (2006) 026 [hepph/0603175].
 (108) T. Sjöstrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten et al., An Introduction to PYTHIA 8.2, Comput. Phys. Commun. 191 (2015) 159 [1410.3012].
 (109) M. Cacciari, G. P. Salam and G. Soyez, The Antik(t) jet clustering algorithm, JHEP 04 (2008) 063 [0802.1189].
 (110) M. Cacciari, G. P. Salam and G. Soyez, FastJet User Manual, Eur. Phys. J. C72 (2012) 1896 [1111.6097].
 (111) L. M. Dery, B. Nachman, F. Rubbo and A. Schwartzman, Weakly Supervised Classification in High Energy Physics, JHEP 05 (2017) 145 [1702.00414].
 (112) T. Cohen, M. Freytsis and B. Ostdiek, (Machine) Learning to Do More with Less, JHEP 02 (2018) 034 [1706.09451].
 (113) G. Blanchard, M. Flaska, G. Handy, S. Pozzi and C. Scott, Classification with asymmetric label noise: Consistency and maximal denoising, Electronic Journal of Statistics 10 (2016) 2780 [1303.1208].
 (114) G. Blanchard and C. Scott, Corrigendum to ‘‘classification with asymmetric label noise: Consistency and maximal denoising’’, Electronic Journal of Statistics 12 (2018) 1779.
 (115) E. M. Metodiev and J. Thaler, Jet Topics: Disentangling Quarks and Gluons at Colliders, Phys. Rev. Lett. 120 (2018) 241602 [1802.00008].
 (116) A. J. Larkoski, D. Neill and J. Thaler, Jet Shapes with the Broadening Axis, JHEP 04 (2014) 017 [1401.2158].
 (117) T. Likhomanenko, P. Ilten, E. Khairullin, A. Rogozhnikov, A. Ustyuzhanin and M. Williams, LHCb Topological Trigger Reoptimization, J. Phys. Conf. Ser. 664 (2015) 082025 [1510.00572].
 (118) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel et al., Scikitlearn: Machine learning in python, Journal of Machine Learning Research 12 (2011) 2825.
 (119) A. J. Larkoski, I. Moult and D. Neill, Toward MultiDifferential Cross Sections: Measuring Two Angularities on a Single Jet, JHEP 09 (2014) 046 [1401.4458].
 (120) M. Procura, W. J. Waalewijn and L. Zeune, Joint resummation of two angularities at nexttonexttoleading logarithmic order, 1806.10622.
 (121) D. Bertolini, P. Harris, M. Low and N. Tran, Pileup Per Particle Identification, JHEP 10 (2014) 059 [1407.6013].
 (122) ‘‘Fastjet contrib.’’ https://fastjet.hepforge.org/contrib/.
 (123) F. Chollet, ‘‘Keras.’’ https://github.com/fchollet/keras, 2015.
 (124) M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean et al., Tensorflow: A system for largescale machine learning., in OSDI, vol. 16, pp. 265283, 2016.
 (125) T. Oliphant, Guide to NumPy. Trelgol Publishing, 2006.
 (126) D. Kingma and J. Ba, Adam: A method for stochastic optimization, 1412.6980.
 (127) DELPHES 3 collaboration, J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens et al., DELPHES 3, A modular framework for fast simulation of a generic collider experiment, JHEP 02 (2014) 057 [1307.6346].