Design and Evaluation of LAr Trigger Digitizer Board in ATLAS Phase-I Upgrade
The LHC upgrade is planned to enhance the instantaneous luminosity during Run 3 from 2021 through 2023. The Phase-I upgrade of the trigger readout electronics for the ATLAS Liquid Argon (LAr) Calorimeters will be installed during the second long shutdown of LHC in 2019-2020. In this upgrade, the so-called super cells are introduced to provide higher granularity, higher resolution and longitudinal shower shape information from the LAr calorimeters to the level-1 trigger processors. A new LAr Trigger Digitizer Board (LTDB) will manipulate and digitize 320 channels of super cell signals, and transmit it via 40 fiber optical links to the back end where data are further processed and transmitted to the trigger processors. Five pairs of bidirectional GBT links are used for slow control from the Front-end LInks eXchange (FELIX) in the ATLAS TDAQ system. LTDB also outputs 64 summed analog signals to the current Tower Builder Board via the new baseplane. A test system is developed to test all functions of the LTDB and carry out the performance measurement. A back end PCIe card is designed which has the circuit to interface to the ATLAS trigger, time and control system. It can control the generation of injection signals to the LTDB for performance test. It also configures and calibrate all ASICs on the LTDB.
ATLAS Liquid Argon (LAr) calorimeter consists of the electromagnetic barrel, the electromagnetic end-cap, the hadronic end-cap and the forward calorimeter. The position of these calorimeters are shown in the Figure 1. In current LAr trigger readout, for each area of size , the Layer Sum Board (LSB), as mezzanines on the front-end board will sum and get the energy deposition across each of the four longitudinal layers in the calorimeter. As depicted in Figure 2, the so called Tower Builder Board will further sum these four energies together, and form a trigger tower with the granularity of .
The second long shutdown of LHC is scheduled in 2019-2020. For LAr calorimeters of the ATLAS detector, the Phase-I upgrade of the trigger readout electronics will be installed. The objective of this upgrade is to provide higher granularity, higher resolution and longitudinal shower information. As shown in the Figure 2, after the upgrade, the level-1 trigger granularity will be improved. One current trigger tower will has 10 so called super cells. The information from each layer is retained, and the granularity can be fine up to . There will be about 34000 super cells in total. All of them will be sampled at every LHC bunch-crossing at a frequency of 40 MHz.
As the LHC luminosity increases above the LHC design value, the improved calorimeter trigger electronics will allow ATLAS to deploy more sophisticated algorithms, enhancing the ability to measure the properties of the newly discovered Higgs boson and the potential for discovering physics beyond the standard model.
As shown in the architecture of the Figure 3, LSBs need to be upgraded to output the super cell signals. The new LAr Trigger Digitizer Boards (LTDB) will process and digitize the super cell signals, and send the processed data to the back-end electronics LAr Digital Processing System (LDPS), where data are then transmitted to the trigger processors. Each LTDB will be able to process up to 320 super cell signals, which will be digitized by 80 of 12-bit quad-channel NEVIS ADCs on the LTDB. Twenty serializer ASICs LOCx2 will receive the ADC data, and output 40 of 5.12 Gb/s data via fiber optical links to the LDPS. The LDPS will convert the samples to calibrated energies in real-time and interface to the FEX processors. With a total of 124 LTDB boards in the system, the total rate to the back end electronics is approximately 25 Tbps. The LTDB will also output 64 summed signals to the tower builder board, each of the signal is sum of 4 input in the same middle or front layer.
The control and monitoring of LTDB is realized via 5 GBT links connected with the Front-End LInks eXchange (FELIX) in ATLAS TDAQ system. FELIX will distribute the TTC (Time, Trigger and Control) clock and BCR (Bunch Counter Reset) signal via down-links to the LTDB. Besides TTC information, FELIX will also control GBTx and all ASICs via the SCA (Slow Control Adapter) ASIC on the LTDB.
Ii Design and Test
Ii-a Design of the LTDB
The LTDB design have been split into three stages. In the demonstrator stage, forty 8-channel TI ADC ADS5272 digitize the 320 super cells. Four Kintex-7 FPGA are used to packing the data and send it to the back-end via 40 of 4.8 Gb/s links. Radiation tolerance of COTS ADCs and power converters are researched. For the pre-prototype, 80 of the NEVIS ADCs are used, 10 of Xilinx Artix-7 FPGA are used for data packing and transmission. From the prototype stage, all of the ADC, serilizer, optical-electric converters are custom radiation-hard ASICs. Diagram of the LTDB prototype is shown in the Figure 4.
The prototype board is shown in Figure 5. Analog section is at the bottom half of board. Digital part is at the top side.
The slow control between FELIX and LTDB are realized via 5 GBT links. The GBT link is connected to GBTx via the MTRx module. GBTx supports to output two kinds of recovered clock: DCLK with better quality and CLKDES with programmable phase in step of 48.8 ps. For 40MHz, the jitter in a frequency range from 100 Hz to 5MHz is about 4 ps for DCLK. For CLKDES, it is about 10 ps. The high quality DCLK is used as the ADC input clock. On the prototype board, LOCx2 also uses DCLK. On the pre-production board, the CLKDES is used to support the phase calibration required by the LOCx2.
Ii-B Test Setup and Results
To test the LTDB boards, as shown in the test setup of Figure 6, the PCIe card BNL-711 is developed. This 16-lane Gen 3 PCIe card can interface the ATLAS TTC system, and decode the TTC clock and TTC information like BCR. There are 48 bi-directional optical links which can run up to 14 Gb/s. The two DDR4 modules can support 32 GB buffer with a speed of up to about 270 Gbps.
In the test setup, the firmware has 6 bidirectional GBT links and 40 GTH receiver for the data links from the LTDB. The low-level software tools from the FELIX project are used to read and write registers in the Kintex Ultrascale FPGA on BNL-711. An optimized low latency GBT-FPGA core is developed to support multi-channel GBT links with LTDB. The IC/EC API is developed to control the GBTx and communicate with GBT-SCA chips via the GBT links. With this API, all ASICs on the LTDB can be configured and calibrated. The high level software is designed to monitor voltage, current and temperature of the LTDB. Beside the five GBT links for LTDB, one extra link is connected to the Xilinx ZC706 evaluation board. This link is used control the DAC3484 evaluation card, which can generate test super cell waveform for LTDB. The Xilinx all digital VCXO PICXO is implemented in ZC706 to synchronize the GBT link to BNL-711 with system clock from BNL-711. The BNL-711 PCIe card are shown in Figure 7.
Iii Summary and outlook
The new LAr calorimeter trigger readout system is being designed for the Phase-I upgrade. LTDB is the kernel part in the front-end. Two LTDB pre-production boards have been installed on the detector in 2018 successfully. The preliminary test results show that the total noise level of the crate with LTDB installed is at the same level of other crates. The pedestal and noise level of super cells are same with test at lab. The data taking is ongoing for the LHC Run 2 with the BNL-711 PCIe card. Full integration with LDPS, FELIX and level 1 calorimeter trigger system is scheduled in summer of 2018.
-  ATLAS Collaboration, The ATLAS Experiment at the CERN Large Hadron Collider, 2008 JINST 3 S08003
-  ATLAS Collaboration, ATLAS Liquid Argon Calorimeter Technical Design Report, CERN-LHCC-96-041, https://cds.cern.ch/record/331061
-  ATLAS Collaboration, ATLAS Liquid Argon Calorimeter Phase-I Upgrade Technical Design Report, CERN-LHCC-2013-017, https://cds.cern.ch/record/1602230
-  J. Kuppambatti et al., A radiation-hard dual channel 4-bit pipeline for a 12-bit 40 MS/s ADC 188 prototype with extended dynamic range for the ATLAS Liquid Argon Calorimeter readout 189 electronics upgrade at the CERN LHC, 2013 JINST 8 P09008
-  L. Xiao et al., LOCx2, a low-latency, low-overhead, 2 Ã 5.12-Gbps transmitter ASIC for the ATLAS Liquid Argon Calorimeter trigger upgrade, 2016 JINST 11 C02013
-  J. Anderson et al., FELIX: a High-Throughput Network Approach for Interfacing to Front End Electronics for ATLAS Upgrades, Journal of Physics: Conference Series, Volume 664, Online computing, 082050
-  J. Anderson et al., FELIX: a PCIe based high-throughput approach for interfacing front-end and trigger electronics in the ATLAS Upgrade framework, 2016 JINST 11 C12023
-  K. Chen et al., Evaluation of commercial ADC radiation tolerance for accelerator experiments, 2015 JINST 10 P08009
-  H. Liu et al., Development of an ADC Radiation Tolerance Characterization System for the Upgrade of the ATLAS LAr Calorimeter, 2017 Chinese Physics C 41 026101
-  CERN GBT Project: GBTX Manual, https://espace.cern.ch/GBT-Project/GBTX/Manuals/gbtxManual.pdf
-  K. Chen et al., Optimization on fixed low latency implementation of the GBT core in FPGA, 2017 JINST 12 P07011
K. Chen. HDLC-ICEC: the first release, 2016 Zenodo,
-  Xilinx. All Digital VCXO Replacement for Gigabit Transceiver Applications (7 Series/Zynq-7000) https://www.xilinx.com/support/documentation/application_notes/xapp589-VCXO.pdf