Design and Evaluation of LAr Trigger Digitizer Board in ATLAS Phase-I Upgrade

Design and Evaluation of LAr Trigger Digitizer Board in ATLAS Phase-I Upgrade

Kai Chen1, Hucheng Chen1, Mauro Citterio3, Herve Deschamps2, Aude Grabas2, Stefano Latorre3,
Massimo Lazzaroni3, Hongbin Liu1, Phillippe Schwemling2, Stefan Simion4, Hao Xu1 and Heling Zhu5,
1Physics Department, Brookhaven National Laboratory, Upton, NY, USA 2CEA-Saclay, Paris, France 3INFN Milano, Milan, Italy 4LAL, Orsay, France 5University of Science and Technology of China, Hefei, China

The LHC upgrade is planned to enhance the instantaneous luminosity during Run 3 from 2021 through 2023. The Phase-I upgrade of the trigger readout electronics for the ATLAS Liquid Argon (LAr) Calorimeters will be installed during the second long shutdown of LHC in 2019-2020. In this upgrade, the so-called super cells are introduced to provide higher granularity, higher resolution and longitudinal shower shape information from the LAr calorimeters to the level-1 trigger processors. A new LAr Trigger Digitizer Board (LTDB) will manipulate and digitize 320 channels of super cell signals, and transmit it via 40 fiber optical links to the back end where data are further processed and transmitted to the trigger processors. Five pairs of bidirectional GBT links are used for slow control from the Front-end LInks eXchange (FELIX) in the ATLAS TDAQ system. LTDB also outputs 64 summed analog signals to the current Tower Builder Board via the new baseplane. A test system is developed to test all functions of the LTDB and carry out the performance measurement. A back end PCIe card is designed which has the circuit to interface to the ATLAS trigger, time and control system. It can control the generation of injection signals to the LTDB for performance test. It also configures and calibrate all ASICs on the LTDB.

I Introduction

ATLAS Liquid Argon (LAr) calorimeter consists of the electromagnetic barrel, the electromagnetic end-cap, the hadronic end-cap and the forward calorimeter[1][2]. The position of these calorimeters are shown in the Figure 1. In current LAr trigger readout, for each area of size , the Layer Sum Board (LSB), as mezzanines on the front-end board will sum and get the energy deposition across each of the four longitudinal layers in the calorimeter. As depicted in Figure 2, the so called Tower Builder Board will further sum these four energies together, and form a trigger tower with the granularity of .

Fig. 1: ATLAS Liquid Argon calorimeter

The second long shutdown of LHC is scheduled in 2019-2020. For LAr calorimeters of the ATLAS detector, the Phase-I upgrade of the trigger readout electronics will be installed. The objective of this upgrade is to provide higher granularity, higher resolution and longitudinal shower information[3]. As shown in the Figure 2, after the upgrade, the level-1 trigger granularity will be improved. One current trigger tower will has 10 so called super cells. The information from each layer is retained, and the granularity can be fine up to . There will be about 34000 super cells in total. All of them will be sampled at every LHC bunch-crossing at a frequency of 40 MHz.

Fig. 2: Geometrical representation in , space of an electromagnetic trigger tower in the current system, where the transverse energy in all four layers are summed (left) and of the super-cells for the Phase-I upgrade, where the transverse energy in each layer is retained in addition to the finer granularity in the front and middle layers (right). Each big square here represents an area of size .[3]

As the LHC luminosity increases above the LHC design value, the improved calorimeter trigger electronics will allow ATLAS to deploy more sophisticated algorithms, enhancing the ability to measure the properties of the newly discovered Higgs boson and the potential for discovering physics beyond the standard model.

Fig. 3: Schematic block diagram of the upgraded LAr trigger readout architecture: with new components indicated by the red outlines and arrows.

As shown in the architecture of the Figure 3, LSBs need to be upgraded to output the super cell signals. The new LAr Trigger Digitizer Boards (LTDB) will process and digitize the super cell signals, and send the processed data to the back-end electronics LAr Digital Processing System (LDPS), where data are then transmitted to the trigger processors. Each LTDB will be able to process up to 320 super cell signals, which will be digitized by 80 of 12-bit quad-channel NEVIS ADCs[4] on the LTDB. Twenty serializer ASICs LOCx2 will receive the ADC data, and output 40 of 5.12 Gb/s data via fiber optical links to the LDPS[5]. The LDPS will convert the samples to calibrated energies in real-time and interface to the FEX processors. With a total of 124 LTDB boards in the system, the total rate to the back end electronics is approximately 25 Tbps. The LTDB will also output 64 summed signals to the tower builder board, each of the signal is sum of 4 input in the same middle or front layer.

The control and monitoring of LTDB is realized via 5 GBT links connected with the Front-End LInks eXchange (FELIX) in ATLAS TDAQ system[6][7]. FELIX will distribute the TTC (Time, Trigger and Control) clock and BCR (Bunch Counter Reset) signal via down-links to the LTDB. Besides TTC information, FELIX will also control GBTx and all ASICs via the SCA (Slow Control Adapter) ASIC on the LTDB.

Ii Design and Test

Ii-a Design of the LTDB

The LTDB design have been split into three stages. In the demonstrator stage, forty 8-channel TI ADC ADS5272 digitize the 320 super cells. Four Kintex-7 FPGA are used to packing the data and send it to the back-end via 40 of 4.8 Gb/s links. Radiation tolerance of COTS ADCs and power converters are researched[8][9]. For the pre-prototype, 80 of the NEVIS ADCs are used, 10 of Xilinx Artix-7 FPGA are used for data packing and transmission. From the prototype stage, all of the ADC, serilizer, optical-electric converters are custom radiation-hard ASICs. Diagram of the LTDB prototype is shown in the Figure 4.

Fig. 4: Diagram of the LTDB prototype

The prototype board is shown in Figure 5. Analog section is at the bottom half of board. Digital part is at the top side.

Fig. 5: LTDB prototype

The slow control between FELIX and LTDB are realized via 5 GBT links. The GBT link is connected to GBTx via the MTRx module. GBTx supports to output two kinds of recovered clock: DCLK with better quality and CLKDES with programmable phase in step of 48.8 ps. For 40MHz, the jitter in a frequency range from 100 Hz to 5MHz is about 4 ps for DCLK. For CLKDES, it is about 10 ps. The high quality DCLK is used as the ADC input clock. On the prototype board, LOCx2 also uses DCLK. On the pre-production board, the CLKDES[10] is used to support the phase calibration required by the LOCx2.

Ii-B Test Setup and Results

To test the LTDB boards, as shown in the test setup of Figure 6, the PCIe card BNL-711 is developed. This 16-lane Gen 3 PCIe card can interface the ATLAS TTC system, and decode the TTC clock and TTC information like BCR. There are 48 bi-directional optical links which can run up to 14 Gb/s. The two DDR4 modules can support 32 GB buffer with a speed of up to about 270 Gbps.

Fig. 6: Test setup for LTDB evaluation

In the test setup, the firmware has 6 bidirectional GBT links and 40 GTH receiver for the data links from the LTDB. The low-level software tools from the FELIX project are used to read and write registers in the Kintex Ultrascale FPGA on BNL-711. An optimized low latency GBT-FPGA core is developed to support multi-channel GBT links with LTDB[11]. The IC/EC API is developed to control the GBTx and communicate with GBT-SCA chips via the GBT links[12]. With this API, all ASICs on the LTDB can be configured and calibrated. The high level software is designed to monitor voltage, current and temperature of the LTDB. Beside the five GBT links for LTDB, one extra link is connected to the Xilinx ZC706 evaluation board. This link is used control the DAC3484 evaluation card, which can generate test super cell waveform for LTDB. The Xilinx all digital VCXO PICXO[13] is implemented in ZC706 to synchronize the GBT link to BNL-711 with system clock from BNL-711. The BNL-711 PCIe card are shown in Figure 7.

Fig. 7: BNL-711 PCIe card for the LTDB test

With this test setup, all functions of the LTDB pre-production are verified. With the successful testing, two LTDB pre-production boards are installed on the detector in early 2018. The pedestal and noise distribution of all channels on one board are shown in the Figure 8 and 9.

Fig. 8: Pedestal distribution of the 320 channels
Fig. 9: Std noise distribution of the 320 channels

Iii Summary and outlook

The new LAr calorimeter trigger readout system is being designed for the Phase-I upgrade. LTDB is the kernel part in the front-end. Two LTDB pre-production boards have been installed on the detector in 2018 successfully. The preliminary test results show that the total noise level of the crate with LTDB installed is at the same level of other crates. The pedestal and noise level of super cells are same with test at lab. The data taking is ongoing for the LHC Run 2 with the BNL-711 PCIe card. Full integration with LDPS, FELIX and level 1 calorimeter trigger system is scheduled in summer of 2018.


  • [1] ATLAS Collaboration, The ATLAS Experiment at the CERN Large Hadron Collider, 2008 JINST 3 S08003
  • [2] ATLAS Collaboration, ATLAS Liquid Argon Calorimeter Technical Design Report, CERN-LHCC-96-041,
  • [3] ATLAS Collaboration, ATLAS Liquid Argon Calorimeter Phase-I Upgrade Technical Design Report, CERN-LHCC-2013-017,
  • [4] J. Kuppambatti et al., A radiation-hard dual channel 4-bit pipeline for a 12-bit 40 MS/s ADC 188 prototype with extended dynamic range for the ATLAS Liquid Argon Calorimeter readout 189 electronics upgrade at the CERN LHC, 2013 JINST 8 P09008
  • [5] L. Xiao et al., LOCx2, a low-latency, low-overhead, 2 × 5.12-Gbps transmitter ASIC for the ATLAS Liquid Argon Calorimeter trigger upgrade, 2016 JINST 11 C02013
  • [6] J. Anderson et al., FELIX: a High-Throughput Network Approach for Interfacing to Front End Electronics for ATLAS Upgrades, Journal of Physics: Conference Series, Volume 664, Online computing, 082050
  • [7] J. Anderson et al., FELIX: a PCIe based high-throughput approach for interfacing front-end and trigger electronics in the ATLAS Upgrade framework, 2016 JINST 11 C12023
  • [8] K. Chen et al., Evaluation of commercial ADC radiation tolerance for accelerator experiments, 2015 JINST 10 P08009
  • [9] H. Liu et al., Development of an ADC Radiation Tolerance Characterization System for the Upgrade of the ATLAS LAr Calorimeter, 2017 Chinese Physics C 41 026101
  • [10] CERN GBT Project: GBTX Manual,
  • [11] K. Chen et al., Optimization on fixed low latency implementation of the GBT core in FPGA, 2017 JINST 12 P07011
  • [12] K. Chen. HDLC-ICEC: the first release, 2016 Zenodo,
  • [13] Xilinx. All Digital VCXO Replacement for Gigabit Transceiver Applications (7 Series/Zynq-7000)
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description