Evolution of the observed Ly\alpha luminosity function from z=6.5 to z=7.7 : evidence for the epoch of reionization ? Based on observations collected at the European Organisation for Astronomical Research in the Southern Hemisphere (ESO), Chile, Prog-Id 181.A-0485, 181.A-0717, 60.A-9284, 084.A-0749. Based on observations obtained at the Canada-France-Hawaii Telescope (CFHT) which is operated by the National Research Council (NRC) of Canada, the Institut National des Sciences de l’Univers of the Centre National de la Recherche Scientifique of France (CNRS), and the University of Hawaii. This work is based in part on observations obtained with MegaPrime/MegaCam, a joint project of CFHT and CEA/DAPNIA and in part on data products produced at TERAPIX and the Canadian Astronomy Data Centre as part of the Canada-France-Hawaii Telescope Legacy Survey, a collaborative project of NRC and CNRS. This paper includes data gathered with the 6.5 metre Magellan Telescopes located at Las Campanas Observatory, Chile.

Evolution of the observed Ly luminosity function from to : evidence for the epoch of reionization ? thanks: Based on observations collected at the European Organisation for Astronomical Research in the Southern Hemisphere (ESO), Chile, Prog-Id 181.A-0485, 181.A-0717, 60.A-9284, 084.A-0749. Based on observations obtained at the Canada-France-Hawaii Telescope (CFHT) which is operated by the National Research Council (NRC) of Canada, the Institut National des Sciences de l’Univers of the Centre National de la Recherche Scientifique of France (CNRS), and the University of Hawaii. This work is based in part on observations obtained with MegaPrime/MegaCam, a joint project of CFHT and CEA/DAPNIA and in part on data products produced at TERAPIX and the Canadian Astronomy Data Centre as part of the Canada-France-Hawaii Telescope Legacy Survey, a collaborative project of NRC and CNRS. This paper includes data gathered with the 6.5 metre Magellan Telescopes located at Las Campanas Observatory, Chile.

B. Clément Laboratoire d’Astrophysique de Marseille, OAMP, Université Aix-Marseille & CNRS, 38 rue Frédéric Joliot Curie, 13388 Marseille cedex 13, France Steward Observatory, University of Arizona, 933 N. Cherry Ave, Tucson, AZ 85721, USA    J.-G. Cuby Laboratoire d’Astrophysique de Marseille, OAMP, Université Aix-Marseille & CNRS, 38 rue Frédéric Joliot Curie, 13388 Marseille cedex 13, France    F. Courbin Laboratoire d’astrophysique, École Polytechnique Fédérale de Lausanne (EPFL), Observatoire de Sauverny, 1290 Versoix, Switzerland    A. Fontana INAF Osservatorio Astronomico di Roma, Via Frascati 33,00040 Monteporzio (RM), Italy    W. Freudling European Southern Observatory, Karl-Schwarzschild Strasse, 85748 Garching bei München, Germany    J. Fynbo Dark Cosmology Centre, Niels Bohr Institute, Copenhagen University, Juliane Maries Vej 30, 2100 Copenhagen Ø, Denmark    J. Gallego Departamento de Astrofísica, Facultad de CC. Físicas, Universidad Complutense de Madrid, E-28040 Madrid, Spain    P. Hibon School of Earth and Space Exploration, Arizona State University, Tempe, AZ 85287, USA    J.-P. Kneib Laboratoire d’Astrophysique de Marseille, OAMP, Université Aix-Marseille & CNRS, 38 rue Frédéric Joliot Curie, 13388 Marseille cedex 13, France    O. Le Fèvre Laboratoire d’Astrophysique de Marseille, OAMP, Université Aix-Marseille & CNRS, 38 rue Frédéric Joliot Curie, 13388 Marseille cedex 13, France    C. Lidman Australian Astronomical Observatory, Epping, NSW 1710, Australia    R. McMahon Institute of Astronomy, Madingley Road, Cambridge CB3 0HA, UK    B. Milvang-Jensen Dark Cosmology Centre, Niels Bohr Institute, Copenhagen University, Juliane Maries Vej 30, 2100 Copenhagen Ø, Denmark    P. Moller European Southern Observatory, Karl-Schwarzschild Strasse, 85748 Garching bei München, Germany    A. Moorwood European Southern Observatory, Karl-Schwarzschild Strasse, 85748 Garching bei München, Germany Deceased. Alan Moorwood’s life and scientific achievements are remembered in the September 2011 (No. 145) issue of ESO’s Messenger.    K.K. Nilsson European Southern Observatory, Karl-Schwarzschild Strasse, 85748 Garching bei München, Germany    L. Pentericci INAF Osservatorio Astronomico di Roma, Via Frascati 33,00040 Monteporzio (RM), Italy    B. Venemans European Southern Observatory, Karl-Schwarzschild Strasse, 85748 Garching bei München, Germany    V. Villar Departamento de Astrofísica, Facultad de CC. Físicas, Universidad Complutense de Madrid, E-28040 Madrid, Spain    J. Willis Department of Physics and Astronomy, University of Victoria, Elliot Building, 3800 Finnerty Road, Victoria, BC, V8P 1A1, Canada
Received May 23, 2011; accepted September 19, 2011
Key Words.:
Methods: observational - Techniques: image processing - Galaxies: high-redshift - Galaxies: luminosity function, mass function - early Universe - dark ages, reionization, first stars
offprints: B. Clément
Abstract

Context:

Aims:Ly emitters (LAEs) can be detected out to very high redshifts during the epoch of reionization. The evolution of the LAE luminosity function with redshift is a direct probe of the Ly transmission of the intergalactic medium (IGM), and therefore of the IGM neutral-hydrogen fraction. Measuring the Ly luminosity function (LF) of Ly emitters at redshift therefore allows us to constrain the ionizing state of the Universe at this redshift.

Methods:We observed three 7575 fields with the HAWK-I instrument at the VLT with a narrow band filter centred at 1.06 m and targeting Ly emitters at redshift 7.7. The fields were chosen for the availability of multiwavelength data. One field is a galaxy cluster, the Bullet Cluster, which allowed us to use gravitational amplification to probe luminosities that are fainter than in the field. The two other fields are subareas of the GOODS Chandra Deep Field South and CFHTLS-D4 deep field. We selected LAE candidates from a variety of colour criteria, in particular from the absence of detection in the optical bands.

Results:We do not find any LAE candidates at in down to a narrow band AB magnitude of 26, which allows us to infer robust constraints on the Ly LAE luminosity function at this redshift.

Conclusions:The predicted mean number of objects at , derived from somewhat different luminosity functions of Hu et al. (2010), Ouchi et al. (2010), and Kashikawa et al. (2011) are 2.5, 13.7, and 11.6, respectively. Depending on which of these luminosity functions we refer to, we exclude a scenario with no evolution from to at 85% confidence without requiring a strong change in the IGM Ly transmission, or at 99% confidence with a significant quenching of the IGM Ly transmission, possibly from a strong increase in the high neutral-hydrogen fraction between these two redshifts.

Received …/ Accepted …

1 Introduction

Observing high-z galaxies within the first billion years of the Universe is one of the main frontiers in extragalactic astronomy. Since the discovery, less than a decade ago, of the first astrophysical object at a redshift above 6, a Ly emitter at redshift 6.56 (Hu et al. 2002), spectacular progress has been made in assembling large samples of high-redshift objects. The two main techniques for finding high-redshift galaxies is to look either for strong absorption breaks in the Ly forest in broad band photometry (Lyman break galaxies – LBGs) or for a photometric excess in narrow band (NB) filters due to the Ly line (Ly emitters – LAEs). In the latter case, the NB filters are usually selected to coincide with regions of low OH emission of the night sky, leading to discrete redshift values. SuprimeCam on the Subaru Telescope has revolutionized the field by enabling large samples of LAEs to be furnished at and (Ouchi et al. 2010; Hu et al. 2010, and references therein). The largest samples of LBGs have been recently assembled (Bouwens et al. 2011) from HST observations after the successful installation of the Wide Field Camera3 (WFC3) in May 2009, but LBGs can also be found from the ground with 8-10 m telescopes equipped with efficient near infrared (NIR) cameras (Castellano et al. 2010). Quasars at high-redshift are also found using the Lyman break technique in multi-colour datasets over very wide fields. Most of the quasars at have been discovered in the Sloan Digital Sky Survey (Fan et al. 2006) and from a targeted programme at CFHT (Willott et al. 2010). Finally, a few gamma ray bursts (GRBs) have been discovered at very high redshift (see e.g. Tanvir et al. 2009, for an example of a GRB at redshift 8.2), nicely complementing the other methods by probing the faint end of the luminosity function.

Combined with observations of the cosmic microwave background (CMB), the recent discovery of large samples of objects at high redshift allows astronomers to build a comprehensive picture of the Universe during the reionization epoch when it was 500 Myr to 1 Gyr old. Polarization measurements of the CMB from WMAP (Larson et al. 2011) show a large optical depth due to Thompson scattering of electrons in the early Universe, suggesting that the reionization started at . Conversely, the strong increase of the optical depth in the Ly forest of high-redshift quasars (Becker et al. 2001; Fan et al. 2006) above Å is a likely indicator that reionization was mostly complete at a redshift of about 6. How and at what pace the reionization process has taken place in the [6–10] redshift range is more difficult to establish from observations, and is still a matter of debate. A compilation of the most recent results and constraints on the neutral-hydrogen fraction of the Universe between redshifts 5 and 11 from various probes is shown in figure 23 of Ouchi et al. (2010).

It has been proposed for a long time to use the Ly transmission by the intergalactic medium (IGM) as a probe of its ionization state during the reionization epoch (see e.g. Santos 2004), hence the strong emphasis recently put on Ly emission of LBGs and LAEs as more and more of these objects become available. Follow-up observations of high-z LBGs at is now underway to detect the Ly line in emission in spectroscopy. Stark et al. (2011) measure an increasing fraction of LBGs with strong Ly emission from to , and conjecture that Ly emission should remain strong at higher redshifts unless the neutral-hydrogen fraction of the IGM suddently increases. Conversely, Fontana et al. (2010) report a low fraction of Ly emitters in a sample of LBGs. These are preliminary results based on still modest spectroscopic samples, and it is expected that ongoing and new observations will clarify the situation in a near future.

Another observational method of probing the Ly IGM transmission is to study the evolution of the LAE luminosity function (LF) with redshift. Ouchi et al. (2010) and Kashikawa et al. (2006, 2011) infer from their observations that the evolution of the Ly LAE LF between and can be attributed to a reduction of the IGM Ly transmission of the order of 20%, which can in turn be attributed to a neutral-hydrogen fraction of the order of 20% at (see e.g. Ouchi et al. 2010). Various models are elaborated to reproduce this claim, which has generated considerable interest (see e.g. Kobayashi et al. 2010; Dayal et al. 2011; Laursen et al. 2011; Dijkstra & Wyithe 2010). However, the universality of the LF from Kashikawa et al. (2006) and Ouchi et al. (2010) has recently been questioned. Hu et al. (2010) report significantly different LF parameters from the observations and analysis of a spectroscopically confirmed sample of NB selected LAEs. Similarly, Nakamura et al. (2011) report significantly lower number counts that they tentatively attribute to cosmic variance. Differences in the selection criteria and in extrapolations of the spectroscopic samples to photometric samples might partly explain the discrepancies between the various Ly LAE LFs available in the literature: Kashikawa et al. (2011) have carried out extended spectroscopic confirmation of their earlier photometric sample, resulting in luminosity functions closer to the ones of Hu et al. (2010). Cassata et al. (2011) report the results from a pure spectroscopic sample of (mostly) serendipitous Ly emitters found in deep spectroscopic samples with VIMOS at the VLT ; this sample is consistent with a constant LAE luminosity function from to as reported in the literature before the recent results from Hu et al. (2010).

The current situation at is therefore unclear, with somewhat contradictory observational results. This hampers the validation of the reionization models and of our understanding of this key epoch of the Universe. The discrepancy limits how well we can understand reionisation during this key epoch of the Universe. New data at will help in resolving the current contention between observational results, while data at higher redshifts can bring new constraints at still poorly explored redshifts. In view of the strong interest in studying Ly emission at high redshifts, searching LAEs at is underway from various groups (Hibon et al. 2010; Tilvi et al. 2010; Nilsson et al. 2007). Finding objects is not only interesting for probing the reionization epoch, but also for assessing the physical properties of these objects, which in turn allow constraining how and when they formed. Due to the extreme faintness of these very high-redshift objects, deriving their properties can only be done statistically over large samples (Bouwens et al. 2011) or on individual objects that are gravitationally amplified. For instance, Richard et al. (2011) infer a redshift of formation of for a gravitationally amplified object at .

This paper presents new results on the Ly LAE LF at , from observations carried out at the VLT with the HAWK-I instrument. This paper is organized as follows. In section 2 we describe the observations and the data reduction in section 3. In section 4 we describe our selection procedure of the LAE candidates. In section 5 we present the constraints that we infer from our results on the Ly LAE LF, before discussing our results in section 6.

We use AB magnitudes throughout this paper. We assume a flat CDM model with and km .

2 Observations

This work is primarily based on extremely deep NIR imaging data obtained with HAWK-I at the VLT, using an NB filter at 1.06 m (hereafter referred to as NB1060). Thanks to its wide field of view (7575), excellent throughput and image quality, HAWK-I is ideally suited to searching for faint NIR objects such as very high-redshift galaxies. The main data set was obtained through a dedicated ESO large programme between September 2008 and April 2010. In addition, we include in our analysis HAWK-I science verification NB data taken in 2007. We also make use of various optical and NIR broad band data, publicly available and/or from our own large programme.

2.1 Fields

Figure 1: Images of the CFHTLS-D4 (left), GOODS-South (centre), and Bullet Cluster (right) fields as in the final NB1060 image stacks. The inner and outer black contours on the Bullet Cluster image represent the regions where the gravitational amplification is respectively () and () for a source at redshift .

In preparing the proposal, we carefully balanced the relative merits of blank fields and cluster fields. While gravitational amplification of background sources by foreground massive galaxy clusters allows us to probe luminosities that are intrinsically fainter than in the field, this is at the expense of areal coverage due to space distortion. The relative merits of blank and cluster fields depend on the shape of the luminosity function (LF) of the objects that are being searched, and on the properties of the observations such as field of view, integation time and overheads (Maizy et al. 2010; Richard et al. 2008). From the Ly LAE LF at that was available at the time of proposal preparation, we computed that either type of fields should yield approximately the same number of targets, while probing different (unlensed) luminosity ranges. We also analysed the balance between wide-shallow and narrow-deep survey strategies. For a total time of about 100 hrs (in the NB filter only), it was deemed that observing four fields in total would be optimal in terms of high-z LAE yield, while mitigating the effects of cosmic variance. Operational constraints, such as the distribution of the fields in right ascension, were additionally taken into account when selecting the fields. Our selected fields were Abell 1689 (131130, -01°20′35″, J2000) and 1E0657-56 (Bullet Cluster) (065829, -55°57′16″, J2000) for the cluster fields, the northern half of the GOODS-S field (033229, -27°44′42″, J2000) and a subarea of the one square degree CFHTLS-D4 field (221638, -17°35′41″, J2000) for the two blank fields.

For Abell 1689, although an extensively studied field, it proved hard to assemble a consistent multiwavelength dataset covering the full 7575 HAWK-I field of view. This field is therefore not included in the present analysis and it will be analysed separetely. The Bullet Cluster is a massive merging cluster that allowed the first direct empirical proof of the existence of dark matter by the combination of strong and weak-lensing analyses (Clowe et al. 2006; Bradač et al. 2006). Both clusters have well-constrained mass models and provide a lens magnification of at least a factor of 1.2 over 50% of the HAWK-I field of view (see Figure 1). The GOODS-S and CFHTLS-D4 field were chosen for the wealth of multiwavelength data, in particular deep optical data, publicly available. For the CFHTLS-D4 field, we chose the location of the HAWK-I observations where NIR data were available111 From the CFHT WIRCam Deep Survey (WIRDS), see http://terapix.iap.fr/rubrique.php?id_rubrique=261 (Bielby et al., in preparation), and paying attention to avoiding the brightest stars present in this field. Figure 1 shows the finding charts corresponding to our observations inside the CFHTLS-D4, GOODS-S and Bullet Cluster fields.

Table 1 summarizes the various observations made as part of our large programme on each of the three fields considered in the present analysis. Figure 2 shows the overall transmission curves of the HAWK-I broad band and NB filters corresponding to these observations. Table 2 summarizes the main ancillary broad band data used in this work. Our large programme data consists of more than 110 hrs of on-sky integration time, of which 80 hrs are NB1060 data.

Field Filter Exposure time Seeing Limiting magnitude aperture corrected limiting magnitude.
(hrs) (″)
CFHTLS-D4 NB1060 26.7 0.53 26.65
CFHTLS-D4 J 5.0 0.46 26.55
CFHTLS-D4 K 0.83 0.50 24.6
GOODS-S NB1060 31.9 0.58 26.65
GOODS-S J 3.3 0.44 26.55
BULLET CLUSTER Y 6.1 0.59 26.50
BULLET CLUSTER NB1060 24.8 0.55 26.50
BULLET CLUSTER J 6.5 0.49 26.55
BULLET CLUSTER K 3.75 0.45 25.45
222
Table 1: HAWK-I narrow band and broad band observations from our large programme
Field Filter Instrument Seeing Limiting magnitude Reference
(″)
CFHTLS-D4 u* CFHT/Megacam 0.92 27.40 aperture corrected limiting magnitude. 1
CFHTLS-D4 g’ CFHT/Megacam 0.85 28.20 aperture corrected limiting magnitude. 1
CFHTLS-D4 r’ CFHT/Megacam 0.77 28.00 aperture corrected limiting magnitude. 1
CFHTLS-D4 i’ CFHT/Megacam 0.73 27.45 aperture corrected limiting magnitude. 1
CFHTLS-D4 z’ CFHT/Megacam 0.72 26.60 aperture corrected limiting magnitude. 1
CFHTLS-D4 J CFHT/WIRCam 0.70 25.10 aperture corrected limiting magnitude. 2
CFHTLS-D4 H CFHT/WIRCam 0.67 24.70 aperture corrected limiting magnitude. 2
CFHTLS-D4 K CFHT/WIRCam 0.65 24.70 aperture corrected limiting magnitude. 2
GOODS-S B (F435W) HST/ACS 0.12 27.95 aperture corrected limiting magnitude. 3
GOODS-S V (F606W) HST/ACS 0.11 28.10 aperture corrected limiting magnitude. 3
GOODS-S I (F775W) HST/ACS 0.10 27.55 aperture corrected limiting magnitude. 3
GOODS-S Z (F850LP) HST/ACS 0.10 27.25 aperture corrected limiting magnitude. 3
GOODS-S J VLT/ISAAC 0.50 26.00 aperture corrected limiting magnitude. 4
GOODS-S H VLT/ISAAC 0.53 25.35 aperture corrected limiting magnitude. 4
GOODS-S K VLT/ISAAC 0.47 24.65 aperture corrected limiting magnitude. 4
GOODS-S Y VLT/HAWK-I 0.55 27.20 aperture corrected limiting magnitude. 5
GOODS-S NB1060 VLT/HAWK-I 0.70 25.65 aperture corrected limiting magnitude. 6
GOODS-S K VLT/HAWK-I 0.40 26.00 aperture corrected limiting magnitude. 5
BULLET CLUSTER R-Bessel Magellan/IMACS 0.60 28.00 aperture corrected limiting magnitude. 7
BULLET CLUSTER B (F435W) HST/ACS 0.10 27.95 aperture () limiting magnitude. 8
BULLET CLUSTER V (F606W) HST/ACS 0.10 28.15 aperture () limiting magnitude. 8
BULLET CLUSTER I (F775W) HST/ACS 0.10 28.55 aperture () limiting magnitude. 8
BULLET CLUSTER I (F814W) HST/ACS 0.10 28.30 aperture () limiting magnitude. 7
BULLET CLUSTER Z (F850LP) HST/ACS 0.10 27.90 aperture () limiting magnitude. 8
333 \tablebib

(1) CFHTLS T0006 release; (2) WIRDS T0002 release; (3) GOODS Version 2.0 HST ACS Imaging Data; (4) Retzlaff et al. (2010); (5) Castellano et al. (2010); (6) ESO Prog-Id 60.A-9284; (7) Clowe et al. (2006); (8) Gonzalez et al. (2009).

Table 2: Ancillary public and private data used in this paper
Figure 2: Transmission curves of the HAWK-I broad band and narrow band filters corresponding to the observations made as part of our large programme. The inset shows the profile of the NB1060 filter.

2.2 The HAWK-I NB1060 data

HAWK-I is a 7575 NIR (0.97-2.31 m ) imager installed on the ESO VLT UT4. It is equipped with four 2048 2048 pixels Hawaii-2RG detectors, separated by 15″ wide gaps. The pixel scale is 01065. The NB1060 filter has a central wavelength of 1062 nm, a full width at half maximum (FWHM) of 100 Å, and is designed to match a region of low OH emission from the night sky. The filter width samples Ly emission in the redshift range z = [7.70 – 7.78]. A detector integration time of 300 s is used for all the NB1060 images, ensuring background limited performance. Random telescope offsets within a box of 20″ for the blank fields and 25″ for the cluster fields is used for dithering. For each field, the NB1060 data are acquired over two epochs separated by one year, allowing us to discard transient objects that could be detected in a one-epoch stack, and not in the other.

The instrument had a thermal leak at the beginning of the first semester, which approximately doubled the total background in the NB1060 filter. After a technical intervention on the instrument after a few months of operations, the background returned to its nominal value, and only the observations of the CFHTLS-D4 field were affected. We were granted compensatory time that allowed us to recover the expected limiting magnitude but at the expense of unbalanced limiting magnitudes (by 0.5 mag) for the first and second epoch observations. For the two other fields the limiting magnitudes between the two epoch observations are within 0.15 magnitude.

In total, after image selection discarding images with poor image quality or too high background, the final stacks used in this analysis total integration times in the NB1060 filter of 26.7 hrs for the CFHTLS-D4 field, 31.9 hrs for GOODS-S (including science verification data) and 24.8 hrs for the Bullet Cluster (see Table 1).

2.3 Other imaging data

In addition to the NB1060 data, we performed dedicated broad band HAWK-I observations within our large programme to complement, on a case-by-case basis, the broad band data that were available elsewhere (see Table 1).

For the CFHTLS-D4 field, we had access to the very deep CFHTLS optical data and to the moderately deep NIR WIRDS data, but not as deep as our NB1060 data. We therefore took additional and data to improve the detection limit in these bands. For the Bullet Cluster, in the absence of well-established datasets in the optical and NIR bands, particularly over the full HAWK-I field of view, we devoted a significant fraction of the time on this field to get additional data in the , , and bands. We note that the HAWK-I filter bandpass includes, at its very red edge, the NB1060 filter bandpass. For this field we therefore secured a coherent and self-consistent dataset. We also used IMACS BVR images of the field obtained at the Magellan telescope (Clowe et al. 2006). For the GOODS-S field, we devoted a few hours of observations in the band filter to reach a limiting magnitude fainter than that achieved with the public ISAAC images. We also used a very deep -band image of this field from a separate HAWK-I large programme (181.A-0717) led by one of us.

3 Data reduction

All reduced image stacks and ancillary data products (e.g. weight maps, etc.) from our large programme are currently available upon request and will be made public through the ESO archive, as part of the Phase 3 process444http://www.eso.org/sci/observing/phase3.html. We detail in this section the data reduction procedures that have been used to generate these high-level data products.

3.1 Overview

We use a mix of IRAF555IRAF is distributed by the National Optical Astronomy Observatory, which is operated by the Association of Universities for Research in Astronomy, Inc., under cooperative agreement with the National Science Foundation. and AstrOmatic666See http://www.astromatic.net/ routines for the data reduction, allowing us to control the reduction process step by step. All single quadrant frames from the HAWK-I mosaic array are reduced similarly and independently until the very final steps. After a first pass at the sky subtraction, all images are characterized in terms of PSF and photometric quality, allowing us to identify and remove low-quality images that would degrade the final products. Depending on the field, between 3% and 13% of the NB1060 images are discarded, mostly when the image quality is worse than 1″. This is the case for a fair fraction of the GOODS-S science verification images, for some of our large programme images that have been executed but not validated by the service observers, and, more rarely, for some fully validated images. A multi-pass sky subtraction is then performed, improving the quality of the masking of the objects at each pass. Images are then scaled to account for photometric variations and registered to a common reference frame. The final stacks are produced by averaging all images with a rejection algorithm using kappa-sigma clipping. The NB1060 photometric calibration is performed on unsaturated bright stars by interpolating their optical and 2MASS photometric data, following the approach described in Hibon et al. (2010). Finally, the final four stacks for each quadrant are aligned onto a common astrometric reference.

3.2 Data processing

  1. Pre-processing. Dark frames and twilight sky flats are combined into master calibration frames on a nightly basis. There are typically eight dark frames and about 30 twilight sky flatfield frames per night. No attempt is made to correct for the detector’s non-linearity, which, according to the HAWK-I users manual, is below the 1% level at 75% of the detector saturation level. This could affect the accuracy of the photometric calibration performed on bright stars, and this is accounted for in the following.

  2. Background subtraction. The most delicate step in the reduction of NIR data is the sky subtraction. With dithered images, the classical way of estimating the sky at any particular pixel is by building a running sky frame for each science frame. This running sky frame is usually computed as the median of frames around the central science frame to which it is subtracted. Some care is required, however, for this step to be optimally performed. First, the sky background varies, even in the NB1060 filter where the sky consists of a mixture of faint OH lines, sky continuum, and possibly faint thermal background leaking through the wings of the filter at wavelengths close to the detector cutoff. In addition, the background patterns have structures at low spatial frequencies that are changing with time, with the strongest changes occurring when the telescope crosses meridian. This is attributed to the rotation of the telescope pupil with time, with maximum velocity when passing the meridian. Therefore, the images used to generate a sky frame are carefully selected so as to have similar sky background patterns and to be close in time (within fewer than 15 days). The images thus selected are further zeroed to their median levels and normalized to their pixel to pixel standard deviation (with rejection of outliers). For each pixel, the median of its values in each running list of frames is computed, with kappa-sigma clipping for rejecting outliers. In this step, objects are masked (meaning that the values of the pixels where objects are detected are not included in the median determination) to avoid biasing the estimation of the sky toward high values. The masking process is initiated on individual sky-subtracted frames (where only the brightest objects are detected) and then repeated several times on combined stacks as described in step # 4.

  3. Bad pixels removal. Once the initial sky subtraction is performed on each science frame, bad pixel maps are generated on a nightly basis. Here again, we use the fact that with dithered data an object moves across the detector while bad pixels do not. Individual pixel values exceeding of the local standard deviation over more than 70% of the frames in a given night are flagged as bad pixels and replaced by a linear interpolation of the surrounding pixel values along image lines.

  4. Object masking for sky subtraction. After the initial sky subtraction step and bad-pixel removal, the images are registered using a first-order astrometric solution and median-stacked with rejection of outliers. A mask is then generated from all objects detected in this image, together with detector regions of poor cosmetics. This mask is then used to reprocess the sky frames as described in step #2, after which a new image stack is produced. Steps #2 to #4 are typically repeated three to five times until the background around the objects in the final stack is flat. This iterative procedure improves the quality of the sky subtraction, which otherwise results in overestimated sky levels noticeable as dark regions around the bright objects and in larger photometric errors. As in step #2, the final sky frames are subtracted to the central science frames after zeroing to their median values and scaling to their standard deviations. Faint low-frequency sky subtraction residuals may still remain at this stage, which are removed with a bi-cubic-spline interpolation of a meshed background frame generated by SExtractor (Bertin & Arnouts 1996).

  5. Correction of photometric variations. Frame-to-frame scaling factors are derived from the number counts measured on bright and unsaturated stars detected in each individual sky-subtracted frame. These scaling factors account for variations of the atmosphere transparency and/or of the airmass. Between two to ten stars per quadrant frame are typically used and the fluxes derived from the SExtractor MAG_AUTO measurements. For all three fields, the variations of these frame-to-frame scaling factors are below 10% peak-to-peak, with a star-to-star variation within each frame of about 1.5%.

  6. Image registration. A relative astrometric solution is computed for each sky-subtracted frame using Scamp and a fourth-order polynomial fit of bright star positions across the detector plane. All the resulting images are then resampled to a common reference frame with Swarp (Bertin et al. 2002) using a LANCZOS4 interpolation kernel and a pixel scale of 01065. The interpolation introduces correlated noise between pixels, and this is accounted for when computing the signal-to-noise ratio of the object as discussed in section  3.3.2. The accuracy of the image registration is well within one pixel for the whole data set.

  7. Final stacks and weight maps. The final stacks for each quadrant are finally produced by averaging with 4 rejection the individual science frames processed as described. In this process, a map identifying the rejected pixels and a sigma map are produced. In the latter, sigma () is the standard deviation of the input pixel values, excluding the rejected ones, entering into the stacks. The weight maps are then derived by computing .

  8. Absolute photometric calibration. The broad band J and K data taken as part of our main programme are photometrically matched to existing photometric catalogues and images of the fields in these filters. We carefully select stars in our HAWK-I images for the photometric match. For the CFHTLS-D4 field, the stellar samples consist of 67 and 58 stars in the J and K bands. The zeropoints of the HAWK-I images are adjusted to match the photometry of these samples to the photometry of the same stars in the WIRDS data (see section 2.1). This process leaves residuals between 0.03 and 0.05 magnitude rms in the J and K bands, respectively. The corresponding magnitudes are found to be in very good agreement with the photometry of the 19 2MASS stars present in the field, which is no surprise considering that the WIRDS data were calibrated against 2MASS. We check that there are no systematic offsets in the colours of our stellar samples compared to the colours determined from the stellar library of Pickles (1998) and from a variety of stellar spectra models at various temperatures and metallicites (Marigo et al. (2008) and http://stev.oapd.inaf.it/cgi-bin/cmd). For the GOODS-S field, we use the publicly available ISAAC J/H/K catalogue (Retzlaff et al. 2010) to compute the zeropoint of our HAWK-I J-band image. For the Bullet Cluster field, the J-band and K-band images are calibrated from the 2MASS catalogue, leaving residual errors of 0.05 mag rms in both bands.

    For the calibration of the NB1060 data, because photometric standards in narrow band filters do not exist, we perform the calibration directly on the image stacks, following the approach detailed in Hibon et al. (2010). It consists in interpolating the NB1060 stellar photometry from the optical and NIR broad band data. This is justified by the large number of photometric datapoints available in at least two of our fields and by the absence of features at 1.06 m in the infrared spectra of stars of spectral types earlier than M5 – the coldest stars in our samples as determined by fitting their spectral energy distribution with the stellar models mentioned above. In practice the procedure consists in performing an ad hoc cubic spline fitting, for each star, of their magnitudes in all available bands. The magnitudes in the NB1060 band are derived from this fit. The procedure is adjusted according to the broad band data available in each field. We use exactly the same approach for calibrating the Y image in the case of the Bullet Cluster.

    For the CFHTLS-D4 field we use the u*, g’, r’, i’, z’ optical data from the T0006 CFHTLS release and the NIR J, H, and K WIRDS data mentioned above. The selection of stars that are neither too bright nor too faint in any of the available images leaves a sample of 23 objects. The residual error on the determination of the zeropoint from this sample is 0.05 mag rms after rejection of outliers.

    For the GOODS-S field, the optical F435W, F606W, F775W, and F850LP magnitudes are taken from the merged HST/ACS catalogue (version r2.0z) available on the GOODS website. In this field, there are 44 suitable stars, and the process leaves a residual error of 0.06 mag rms in determining the NB1060 zeropoint.

    For the Bullet Cluster we use a slightly modified procedure to calibrate the Y and NB1060 images. This is because of the lack of optical data for the entire field of view covered by HAWK-I, preventing us from performing a robust interpolation based on a large number of stars between the two wavelength ranges. Instead, we empirically determine the NB1060 zeropoints by matching the J vs J NB1060 (resp. K vs NB1060 K) colour-magnitude diagrams of the stars present in this field to the same diagrams produced on the GOODS-S and CFHTLS-D4 fields after calibration. The same procedure is used to calibrate the Y-band image by comparing it to the colours of the stars in the GOODS-S field for which Y band data are available.

    All the procedures described above are conducted quadrant by quadrant, with a further iteration on the full four quadrant images. In total, considering the consistency between the many checks that are performed, and despite the various methods used, we estimate that the final accuracy of the photometric calibration is of the order of 0.1 magnitude rms.

  9. Absolute astrometric registration and final image stitching. The last step in our reduction process consists in stitching and registering the four detector images to the reference images of each field. The final astrometric solution is computed by Scamp with a fourth-order polynomial fit of the star positions. The CFHTLS-D4 stacks are aligned to the archival CFHTLS images. The GOODS-S stacks are aligned to the optical HST/ACS images, and the Bullet Cluster stacks are aligned to the 2MASS catalogue in the absence of astrometrically calibrated data across the entire area. The final astrometric residuals are below the 005 rms level across the entire field of view for all images. Finally, the resampling and final image stitching are performed using Swarp (Bertin et al. 2002). In addition, Swarp propagates the astrometric solution to the weight maps and uses a weighted mean to compute pixel values in the small overlap between quadrants due to the dithering pattern.

3.3 Final image properties

We now discuss the global properties of the final images: image quality (FWHM), noise, and detection limits. For consistency, the same procedure is applied to all the images used in this work, including archival data. The FWHM and the detection limits are listed in Table 1.

3.3.1 Image quality

The image quality is determined from a high signal-to-noise ratio point spread function (PSF) generated by stacking unsaturated and isolated stellar images (range of 20–30) in each field. After normalization, the stellar images are centred and median-stacked with a 4 outlier rejection. The rejection reduces the contribution from faint neighbouring objects in the wings of the PSF but does not affect its profile. The resulting NB1060 images are slightly elongated, for reasons that are unknown to us, with a measured ellipticity from about 0.05 to 0.1 along a direction away from the N-S axis. The FWHM values are derived from a 2D-Gaussian fit to the median profile. The three NB1060 final images have exquisite image qualities ranging from 053 to 058 (see Table 1).

3.3.2 Photometric errors and correlated noise

Image resampling introduced by the distortion correction, shifting, stitching, and registration processes introduces correlation in the noise of the images. This leads in turn to underestimating the photometric errors when considering the pixel-to-pixel noise properties, see e.g. Grazian et al. (2006), Appendix A, or Casertano et al. (2000). To measure and account for this well known effect, we carefully analyse the noise properties of the images over apertures of varying sizes and derive correction factors that we can then apply to the photometric data measured by SExtractor. Indeed, SExtractor derives the photometric error for each object it finds by computing the local pixel-to-pixel noise fluctuation in the vicinity of the object (in the faint-object, background-limited regime). The SExtractor photometric errors are therefore affected by the correlation of the noise. For each image, we select a thousand positions corresponding to source-free background regions determined from the final SExtractor segmentation (object mask) image. For each position, we measure the integrated flux in circular apertures of diameters ranging from = 1 to 25 pixels (01065 to 2663). For a given aperture size, the variance of these fluxes, , differs, because of the correlated noise, from the variance of the errors computed from the pixel-to-pixel variance and the number of pixels in the aperture. The ratio of these two quantities gives the noise-correction factor that can be used to correct the photometric errors measured by SExtractor. The ratio clearly depends on the aperture size: for apertures smaller than the correlation length of the noise, which is related to the size of the resampling interpolation kernels, , whereas for large apertures , ranging from 1.11 for a three-pixel diameter aperture to 1.67 for a 25 pixel diameter aperture. The procedure is repeated ten times for each field, and all three fields give similar and consistent measurements.

This analysis finally allows us to assign signal-to-noise ratios (SNR) to the objects detected by SExtractor, in the sky background limited regime, using

(1)

where is the photometric error measured by SExtractor. The relation between SNR and the magnitude error is finally given by

(2)

3.3.3 Aperture corrections and optimal apertures

We measured curves of growth on unsaturated stars for all image stacks used in this work, using apertures between 1 and 150 pixels (01065 to 16″) in diameter. Less than 1% of the flux resides in the wings of the PSF beyond radii of 7″, and we therefore safely use the 16″aperture correction to estimate the total flux of unresolved or moderately resolved objects. From the curves of growth of both flux and noise, we derived the optimal diameter that maximizes the signal-to-noise ratio for point-like objects. In practice, for all of our NB1060 image stacks, a diameter of (6 pixels) is used. The corresponding aperture corrections for the three NB1060 final images are mag, mag, mag for the GOODS-S, CFHTLS-D4, and Bullet Cluster fields, respectively. The corresponding noise correcting factors are = 1.22, 1.14, and 1.14 for the GOODS-S, CFHTLS-D4, and Bullet Cluster fields, respectively.

3.3.4 Detection limits

Finally, from the two parameters defined above, and , one can define the limiting magnitude for point-like objects:

(3)

where , and correspond to apertures of 064 in diameter. This leads to a NB1060 point source detection limit of , 26.65, and 26.50 for the GOODS-S, CFHTLS-D4, and Bullet Cluster, respectively, as reported in Table 1.

4 Candidate selection

4.1 Detection completeness

Figure 3: Completeness levels of the NB1060 images for the three observed fields. The coloured areas correspond to the Poissonian errors on the number counts in the corresponding magnitude bin. The triangles indicate the 5 magnitude limits.

We use SExtractor (Bertin & Arnouts 1996) version 2.8.6 for source detection and photometric measurements. We fit the PSF median profile discussed in section  3.3.1 with a sum of three 2-dimensional Gaussians from which we derive the filter ( pixels or ) used by SExtractor for spatial filtering during the detection process. We then use this point-source model to generate mock sources injected into the image to estimate the image detection completeness. We inject 500 mock sources per magnitude bins in regions of the images randomly distributed and free of objects. We perform a number of tests to determine the optimum SExtractor detection parameters that maximize the number of detected objects while minimizing the number of false alarms. Considering the absence of candidate LAEs in our images, we choose to push to the faintest possible limits. False alarms are investigated by running SExtractor on the negative images and are discussed in section 4.3. We determine that an adequate set of SExtractor parameters, for the purpose of our analysis, is to trigger a detection on one pixel (DETECT_MINAREA) after spatial filtering at 0.7 above the local background (DETECT_THRESH). With these detection parameters, the average signal-to-noise ratio of sources in the 50% completeness magnitude bin is of about 4, and a SNR of 5 corresponds to a completeness rate of 70 to 80%. Figure 3 shows the completeness rates achieved in the three NB1060 images.

4.2 Selection criteria

We do not expect LAEs to be detected in any of the filters blueward of the Ly line redshifted to 1.06 m. First, negligible amounts of radiation are expected to escape the galaxy and to be transmitted by the IGM below the Lyman limit, which is redshifted to 790 nm. In addition, all the radiation between the Ly and Ly lines at is entirely redshifted beyond the Gunn-Peterson trough at 850 nm observed in the spectra of high-redshift quasars (Fan et al. 2006). Only in the blue part – the most depressed part – of the Ly forest, just above the Lyman limit, can we therefore expect some flux from a LAE to arrive on Earth, in the wavelength range [790–850] nm approximately. In practice, considering the limiting magnitudes of our optical and NB1060 images, an absorption of 2 magnitudes or so will result in no detections in any of the optical bands.

For the purpose of the analysis described in this paper, we built a master catalogue of all the NB1060 detected objects, measuring their magnitudes in each of the optical and NIR broad band images by running SExtractor in double image mode. To do so, we resample all images to the HAWK-I pixel scale of 01065. We then search objects in this master catalogue that are not detected in the optical images at an initial level, reduced to after visual inspection. In the case of the CFHTLS-D4 field, we request in addition that these objects not be detected at a significance level in a image of the field obtained by combining the g’, r’, and i’ images. In the case of the GOODS-S field, we use a similar non-detection limit on a bviz image obtained by combining the four broad band HST/ACS images. Because the NB1060 bandpass is located within the bandpass of the Y filter (at its red edge), the Ly line may be detected in the Y filter. To estimate the Y NB1060 colour as a function of redshift, we generate simple synthetic models of LAE spectra, consisting of a narrow Ly line and a UV-continuum of energy distribution . We allow the UV slope to vary from -3 to 0, and we set the flux density below the Ly line to zero. For objects with redshifts in the interval [] corresponding to the NB1060 filter, we find that the Y NB1060 colour is , with the lowest values corresponding to situations where the Ly line is redshifted near the edges of the NB1060 filter transmission curve, and therefore strongly attenuated. This corresponds to the case of high-redshift LBGs detected through their continuum in the NB1060 filter. As we see in the next section, the Y images (when available) are not deep enough, relative to the NB1060 images, to measure such a colour on the faintest objects, but they do allow us, conversely, to discard blue and moderately bright objects that pass the other colour selection criteria. To secure the presence of an emission line in the NB1060 filter, we further require a 1 NB excess over the flux measured in the J-band. From equation (6) in Hibon et al. (2010), NB1060 J corresponds to equivalent widths EWÅ or EWÅ, assuming a flat continuum spectrum (). We note that the NB1060 filter is placed approximately at the centre of the bandpass covering the Y and J filters, allowing us to further constrain the presence of an LAE from its colour between the NB1060 and Y+J bandpasses; however, in the absence of candidates from the criteria used so far (see next section), this did not prove necessary to add to our selection criteria. Finally, we restrict the analysis to sources having in the NB1060 final images and in partial or intermediate image stacks corresponding to different observing epochs. In summary, our detection criteria are

  1. NB1060 NB1060 NB1060 ,

  2. no detection above the level in any of the visible broad band filters,

  3. Y NB1060 (when Y band data are available),

  4. NB1060 J with significance.

We finally note that astrophysical sources such as transients, extremely red objects (EROS), high-EW low-z line emitters or T-dwarfs can potentially satisfy the optical non-detection criteria defined here. Relatively deep NIR Y and/or J and/or K band data are therefore required for consolidating the selection (e.g. criteria #3 and #4) and reducing contamination from astrophysical sources. The J and band data in particular are useful to identify T-dwarfs and EROs, even if the latter are often detected in deep optical images. In the absence of LAE candidate in our data, we are clearly not affected by contamination, thanks to the coherent datasets that we use. We refer the interested reader to section 3.3 of Hibon et al. (2010) for a somewhat more detailed analysis of contamination effects in a similar LAE search, in particular by , [OIII], and [OII] line emitters.

4.3 Selection field by field

The 5 detection limit in the NB1060 filter corresponds to magnitudes to depending on field (see Table 1), and to a completeness level of 70 to 80% (see figure 3). The corresponding colour criteria used for the selection of candidates differ among the three fields depending on the depth of the optical images available in each of them.

CFHTLS-D4. The selection criterion #2 in the previous section corresponds to the following colour criteria:

(4)

There are 6500 NB1060 objects detected in this field. The application of criteria #1 and #2 of section 4.2 yields 20 objects. Ten are visually identified as instrumental artefacts caused by electronic crosstalk (see section 4.4). Seven of the remaining objects are detected and relatively bright in the J band and therefore rejected after application of criterion #4. We note in passing that the brightest of these objects has , and and is very likely a T-dwarf.

Amongst the three remaining objects, one is located near the edges of the image and appears sharper than the PSF. One is located in the wings (4″) of a bright extended galaxy, therefore of suspicious photometry and therefore unusable. Finally, the last one appears to be a variable, extended object, with magnitude difference between the first- and second-epoch observations. All three objects are therefore discarded.

To allow for possibly slightly extended LAE candidates and for consistency checks, we carry out a second selection using larger apertures (), and applying the appropriate aperture correction. Criteria #1 and #2 yielded 14 detections, out of which nine are in common with the previous sample of 20 objects. Amongst the remaining five new detections, one is an obvious artefact near a bright star, two are detected in the J-band and are rejected due to the low significance of their NB excess. The two last ones are low significance detections in the NB1060 image () and both show extended and dubious morphologies.

Checking the robustness of the rejections further, we investigate the false alarms on the negative image using the two aperture diameters mentioned above. After removing the well-determined crosstalk features, which have a negative component, we are left with eight objects using the aperture and 2 using the aperture, all of them at the limit of our signal-to-noise ratio selection. These genuine noise artefacts have very similar morphologies to those of the positive detections that were rejected on the science image: either very sharp or extended and irregular with bright non-contiguous pixels. This legitimizes our somewhat ad hoc, but pragmatic earlier selection based on the morphology of the faintest positive candidates, and we therefore conclude that there are no LAE candidate in the CFHTLS-D4 field.

GOODS-S. Here the selection criterion #2 corresponds to the following colour criteria:

(5)

There are 5100 NB1060 detected objects. The application of criteria #1 and #2 yields 16 objects, of which 12 are visually identified as instrumental artefacts caused by electronic crosstalk. Amongst the four remaining objects, one is due to a mismatch on a blended object, and two are marginally detected objects at the edges of the image. The last object is detected in the Y-band but does not satisfy criterion #3 above; interestingly, it is identified with reference G2_1408 as a 7 object in Castellano et al. (2010, and references therein). This object has been followed up in spectroscopy (Fontana et al. 2010), yielding a tentative detection of the Ly line at a redshift . This object therefore appears to be an LBG, caught by its strong UV continuum emission detected in the NB1060 filter.

Similar to what was done on the CFHTLS-D4 field, we then performed a second selection using apertures. This yields two new detections (beyond the obvious electronic artefacts): one is detected in the Y-band and does not pass criterion #3, and is also marginally noticeable in the optical bands. The other one has a dubious morphology and is close to a bright object, and is therefore rejected. Finally, we carried out the false alarm analysis on the negative image, and we detect a handful of events with dubious morphologies, leading to the same conclusions as for the CFHTLS-D4 field.

Bullet Cluster. The dataset for the Bullet Cluster field is somewhat different than for the other fields. As explained in section 2.3, we accomodated a consistent set of HAWK-I data in the Y, NB1060, J and K bands within our large programme. In addition, we used HST/ACS images of the inner part of the field, as well as moderately deep IMACS images from the Magellan telescope (Clowe et al. 2006). There are 7000 NB1060 objects satisfying criterion #1, the vast majority of which are detected in the Y filter and do not satisfy criterion #3. Only 127 objects are not detected in the Y image at 3, but because , it is impossible to conclude whether they satisfy criterion #3 or not. After visual inspection and rejection of electronical ghosts and obvious artefacts, all but one object show a clear counterpart in either one of the HST/ACS images or in the IMACS images. This object shows no NB excess (), and together with a marginal detection in the Y-band (, ) and a non-detection in (, upper limit), we conjecture that this object is probably a T-dwarf. Finally, a selection based on larger apertures as for the two other fields yields no new candidates. We therefore conclude, again, that there are no LAE candidates in the Bullet Cluster field.

4.4 Instrumental artefacts

Instrumental artefacts are a potentially important source of contamination. As explained in section 4.3, candidates are found that are rejected as instrumental artefacts. We describe here some of the instrumental artefact sources that are observed in the HAWK-I data.

  1. Electronic crosstalk. The HAWK-I data suffer from inter-channel crosstalk from the readout electronics. This results in donut-shaped artefacts at regularly spaced intervals along detector lines where bright stars are present. These artefacts were largely attenuated after a technical intervention in the instrument that took place in May 2009. The crosstalk pattern follows the dithering pattern and is therefore present on the final stacked images. The crosstalk artefacts are easy to recognize from their shapes and fixed distances from bright stars along detector rows. Because they do not have counterparts in optical images, these artefacts are selected as candidates in our analysis, but are easily dealt with a posteriori. No attempt was made to remove these artefacts during data processing.

  2. Optical ghosts. Reflections inside the instrument generate typical out-of-focus and decentred pupil images around bright stars. The surface brightness of these haloes was measured to be of the peak intensity in the PSF profiles. Only focussed optical ghosts can be mistaken as candidates, and no such artefacts are observed on the HAWK-I images.

  3. Persistence. Persistence from previously observed bright stars is at a fixed detector position. The persistence features therefore do not follow the dithering pattern and are rejected when combining the images with sigma clipping in the final stacks. Considering the large number of frames used in the stacks (more than 200 frames), persistence effects are unlikely to leave residuals that can be mistaken as candidates.

  4. Radioactive events. One of the HAWK-I arrays (chip #2, Id: ESO-Hawaii2RG-chip78) suffers from a strong radioactive event rate, coming from the detector substrate (Finger et al. 2008). These radioactive events generate showers of variable intensity (typically thousands of electrons) and extent (typically a few tens of pixels on a side). Some events can be as bright as a few hundred thousand electrons and extend up to 400 pixels in one direction. Because of the long detector integration times (DIT) used for the NB1060 images (300 seconds), there are a few tens of such events in a single frame. This results in poor background subtraction and moderately high-frequency residuals (a few tens of pixels) in individual sky-subtracted frames. Although the global noise properties are not significantly different in this quadrant than in the others, the overall cosmetics are somewhat poorer. As a consequence, most of the low signal-to-noise ratio detections and with dubious morphologies reported in section 4.3 appear to be in this quadrant and are therefore rejected as artefacts after visual inspection.

  5. Noise. As explained earlier, we chose a low detection threshold in order to push to the faintest detection limits, triggering a handful of low signal-to-noise ratio false alarms, particularly on the highly radioactive quadrant. We settle on a detection threshold that allows us to handle these false alarms.

5 Constraints on the Ly luminosity function

5.1 Comoving volume

The effective field of view in the transverse dimension of the NB1060 stacks is computed as a function of the detection limit for each field from the background noise maps generated by SExtractor, and accounting for the correlation of the noise as described in section 3.3.2. The sensitivity is reduced at the edges of each quadrant image thanks to the dithering process and in regions close to bright objects. All objects are masked, reducing the total effective area by 10% for the blank fields, and 25% for the Bullet Cluster due to the large number of bright galaxy cluster members. For this field, we further use a map of the gravitational amplification and space distortion from a detailed model of the cluster (Richard et al. in preparation) to compute the effective, unlensed, comoving area corresponding to the image (see Willis et al. 2008, for a similar example). In the direction of the line of sight, we use the filter transmission curve to determine the effective width. We then approximate the NB1060 filter with a rectangular filter of width equal to this effective width. The effects of the filter transmission curve on LAE detection and comoving volumes are detailed in e.g. Willis & Courbin (2005) and Hu et al. (2010). Considering our null results, these effects will not affect our conclusions. We also assume that the filter transmission curve does not vary significantly over the instrument’s field of view, an assumption motivated by the relative uniformity of the sky background over the instrument’s field of view. We finally assume that the observed Ly emission lines are significantly narrower than the filter width, as is the case for high-z LAEs, and we therefore ignore the effect of the line width on the line flux measured through the NB1060 filter. The effective width of the NB1060 (Å) defines the [] redshift interval probed by our observations.

Converting magnitudes to line luminosities requires assumptions on the equivalent width (EW) of the Ly line. Distributions of observed Ly lines in high-z LAEs vary from a few tens of Angstroms to lower limits of a few hundred Angstroms (see e.g. Taniguchi et al. 2005; Ouchi et al. 2010). The Taniguchi et al. (2005) distribution of Ly line EWs is consistent with a conversion factor of 70%, as used in Hibon et al. (2010), when assuming that the EW lower limits are the real values. Conversely, Kobayashi et al. (2010) predict a distribution of Ly EWs clearly peaked toward high values, hence favouring conversion factors closer to 100%. In the following, we therefore use these two values (70% and 100%) when converting NB1060 magnitudes to line luminosities.

In total, the comoving volume sampled by our images is for the Bullet Cluster and for each of the two blank fields, corresponding to a grand total of for the three fields. The volume-luminosity relation is shown in figure 4, and the effect of the gravitational amplification for the Bullet Cluster is clearly visible, where the gravitational amplification enables probing fainter luminosities, however over increasingly smaller volumes.

Figure 4: The comoving volume sampled by the NB1060 images as a function of Ly luminosity. The two dashed curves correspond to the two blank fields (green – CFHTLS-D4; red – GOODS-S). The thin black curve corresponds to the Bullet Cluster. The thick black curve is the total comoving volume corresponding to the three fields. The Ly luminosity corresponds to a 5 NB1060 limiting magnitude, assuming a 70% average conversion factor between the Ly luminosities and NB1060 magnitudes (see text).

5.2 Cosmic variance and Poisson noise

We do not detect any LAE candidates down to a NB1060 5 limiting magnitude of 25.9 to 26.1. To constrain the luminosity function of Ly emitters and compare our results with others, we make use of the Schechter formalism (Schechter 1976):

(6)

where is the characteristic luminosity defining the LF high luminosity cutoff, a volume density normalization factor, and the faint-end slope characterizing how steeply the LF increases at low luminosities. Limited samples and large errors lead to degneracy between these three LF parameters. Pending more observational data and more accurate determinations of the high-z LAE LF parameters, most of the authors in the literature have settled on a canonical faint end slope value of . Our observations probe luminosities similar, or slightly fainter, than those observed by other groups; therefore, we can compare our results to others, and to this aim we similarly adopt, unless stated otherwise, a faint end slope of = -1.5.

Denoting the comoving volume probed by our observations as a function of the luminosity , as described in section 5.1 and shown in figure 4, the total number of objects is given by

(7)

where is the completeness function (see section 4.1). The conversion of the completeness function as a function of magnitude as shown in figure 3 to takes into account the conversion factor of 70% and 100% mentioned above, and in the case of the Bullet Cluster further takes the amplification map of the cluster into account.

From a Poisson distribution, one can easily compute single-sided confidence levels (CL) for the upper limits of the expected number of events that correspond to a measured number of events , as (Gehrels 1986):

(8)

In our situation of zero detection (), the 84.13 , 97.72, 99.87, and 99.99 percentiles777Corresponding to the single sided 1,2,3,4 percentiles of a normal distribution confidence levels correspond to upper limits of the mean number of events of 1.84, 3.78, 6.61, and 10.36, respectively. The situation corresponds to a 63% confidence level. Therefore, with zero detection and assuming pure Poisson statistics, one can exclude, at a given confidence level CL, the luminosity function parameters that would yield an expected number of objects with our survey parameters.

However, considering the somewhat limited area covered by our observations, we need to consider the effects of cosmic variance in our statistical analysis. Somerville et al. (2004) are among the first authors to derive quantitative estimates of the effects of cosmic variance from cold dark matter (CDM) models and observations of the two-point correlation functions of galaxy populations in the GOODS survey data. Trenti & Stiavelli (2008) expand on this work and propose a cosmic variance model, based on N-body simulations and halo occupation distribution models, and applied to a variety of high-z galaxy populations. We use below the on-line version of this model to estimate the effects of cosmic variance on our observations.

Various prescriptions have been proposed for the distribution function of galaxy number counts affected by cosmic variance, see Yang & Saslaw (2011) for a recent discussion and analysis of the galaxy counts-in-cells distribution functions in the SDSS data. Although not physically motivated, the negative binomial distribution (NBD) fits the SDSS data well at both low and high number counts and provides a convenient description for the distribution function of galaxies with positive number counts and overdispersion relative to a Poisson distribution. To prevent technical problems with the use of normal or lognormal distributions (e.g. truncation to positive numbers), we chose to adopt the NBD as an ad hoc representation of the probability density function of low galaxy number counts. The NBD can be conveniently expressed as a Poisson random variable whose mean population parameter is itself random and distributed as a Gamma distribution of variance equal to the relative cosmic variance.

By definition of the cosmic variance, the variance of the number of galaxies of mean number is in excess of the Poisson variance and is given by

(9)

where is the relative cosmic variance. With the NBD prescription, one can derive the confidence level CL corresponding to no detections in our observations, for a known . To estimate the cosmic variance, we run the on-line cosmic variance calculator888http://casa.colorado.edu/trenti/CosmicVariance.html (Trenti & Stiavelli 2008) and derive for low number counts . We use the parameters corresponding to our observations: the instrument’s field of view, a mean redshift of 7.74, and a redshift interval of 0.08. We select the Sheth-Tormen bias formalism, a value of 0.8 for , and a value of 1.0 for the user-selectable halo-filling factor. Although the filling factor of LAEs is likely to be significantly lower than 1.0, see e.g. Ouchi et al. (2010), adopting a value of 1.0 leads to overestimating the importance of cosmic variance and therefore corresponds to a worst-case scenario in our analysis. Finally, we define a completeness factor of 1.0 in the cosmic variance calculator since we use our measured completeness function in equation 7 to derive the number of galaxies that correspond to a given set of LF parameters. We then fit the resulting relative cosmic variance corresponding to our survey parameters as a function of , and we plug this cosmic variance into a Gamma distribution. For each pair of () parameters we thus derive a confidence level corresponding to no detections in our observations.

This is represented in figure 5 together with the best-fit parameters of the Ly LAE luminosity functions at redshifts 5.7 and 6.5 from various sources. We report in this plot the range of () parameters excluded at 85% and 99% confidence levels, for conversion factors between NB1060 magnitudes and Ly luminosities varying from 70% and 100% (see discussion in section 5.1). We report the upper () exclusion zones derived from the absence of detections in the HAWK-I observations only (this work). Furthermore, we report the upper exclusion zones at the same confidence levels when adding to the null HAWK-I detections the null spectroscopic confirmation of the five brightest LAE candidates999Indeed, spectroscopic follow-up with X-shooter at the VLT of the five brightest candidates failed to detect Ly emission. These results will be reported separately in Cuby et al. (in prep.). presented in Hibon et al. (2010). Finally, for illustration, we also report the 85% confidence level derived from this work (HAWK-I observations only) for a faint end slope of the Ly LAE LF .

Figure 5: Parameters of the luminosity function excluded at 85% and 99% confidence levels from our data, assuming the Schechter formalism and a fixed faint end slope unless otherwise stated. Filled circles correspond to the best fit LF parameters at (blue) and (red) from Kashikawa et al. (2011). The ellipses correspond to the 3 confidence levels for these datapoints. Open circles correspond to the best fit LF parameters of the photometric samples of Ouchi et al. (2008) at (magenta) and (blue) and of Ouchi et al. (2010) at (red). The ellipses correspond to the 2 confidence levels for these datapoints. The filled square symbols correspond to the best fit LAE LF parameters at (blue) and (red) from Hu et al. (2010). The plain (resp. dotted) black lines correspond to the 85% and 99% confidence levels corresponding to no detections in our HAWK-I data (this work), assuming a conversion factor of 70% (resp. 100%) between the NB1060 and Ly fluxes. The dark (resp. light) grey zones correspond to the range of 85% (resp. 99%) confidence levels for conversion factors between 70% and 100%, as delimited by the plain and dotted lines. Similarly, the lines and hatched zones in magenta colour correspond to the 85% and 99% confidence levels when adding to our work the null spectroscopic confirmation of the five brightest LAE candidates of Hibon et al. (2010).

6 Discussion

The various LF parameters reported in the literature at redshifts 5.7 and 6.5 present some significant differences. We report in figure 5 some of the best-fit Schechter parameters at redshifts 3.1, 5.7 and 6.5. We report photometric samples partially confirmed in spectroscopy and purely spectroscopic samples. At the same redshift, the two series of datapoints clearly differ by significant amounts.

Ouchi et al. (2010), and previously Kashikawa et al. (2006) followed by Kashikawa et al. (2011), convincingly claim that the evolution, mostly in luminosity, of the LF from the lower redshift LFs at and LFs is a signature of reionization, due to a neutral-hydrogen fraction of the order of 20% at . Conversely, the Hu et al. (2010) evolution of the LF parameters between the two same redshifts is mostly in density; accordingly, they do not infer a signature of reionization, in agreement with an earlier claim by Malhotra & Rhoads (2004) that there was no evidence of reionization in the evolution of the LAE LF between these two redshifts.

In the light of the recent work from Hu et al. (2010), the question of the evolution of the Ly LAE luminosity function beyond redshift 5.7 becomes more difficult to comprehend. If the point of Hu et al. (2010) (and Malhotra & Rhoads 2004) is correct, the picture by which the LAE LF would be approximately constant between and , and would evolve significantly beyond redshift as a consequence of an increasingly neutral intergalactic medium (IGM), might need to be revisited. How useful are our results in this context?

From figure 5, we infer that we can safely exclude a non evolution of the LF parameters at at more than 99% confidence level from the Kashikawa et al. (2006, 2011), Ouchi et al. (2008) and Ouchi et al. (2010) values at and 6.5. In that case, we should find and LAEs in the three HAWK-I fields based on the LF parameters estimated by Ouchi et al. (2010) and Kashikawa et al. (2011) at , respectively. The LF parameters of Hu et al. (2010) would predict LAEs at and can be similarly excluded at an confidence level, or higher, when including the null spectroscopic confirmation of the brightest LAE candidates of Hibon et al. (2010) (see previous section and footnote 9). Our results therefore clearly show that the Ly LAE LF does evolve from lower redshifts, but in the absence of concordance between the data at lower redshifts, it is difficult to ascribe this evolution to the galaxy properties or to reionization.

To continue the discussion, we now consider two scenarios for the evolution of the LAE LF between and , either in density or in luminosity. These scenarios are purely phenomenological and are not supported by theoretical considerations, only serving the purpose of assessing what the consequences of our results might be. This is illustrated in figure 6 in a cumulative luminosity function plot. For a luminosity function to be consistent with our null results, it should lie, ignoring statistical fluctuations, below the contours corresponding to the parameter space probed by our observations.

In the first scenario, we consider a 60% evolution in density from the datapoint of Hu et al. (2010). Such an evolution can be entirely ascribed, in principle, to an intrinsic evolution of the LAE LF between and as in the models of Kobayashi et al. (2007) where the LF, multipling by 0.4, almost perfectly coincides with the LF. In practice, such an evolution could result from the combined evolution of the density of dark matter haloes and of the intrinsic galaxy properties. We also note that the UV LF of high-redshift galaxies between and evolves essentially in luminosity (Bouwens et al. 2011), so a pure evolution in density should be treated with caution. As can be seen in Figs. 5 and 6, such an evolution would fit most of the observational data, the shapes of the lower-redshift LFs, the datapoint from Iye et al. (2006) at , the datapoints from Vanzella et al. (2011) at and , and our data. The mere existence of the Iye et al. (2006) and Vanzella et al. (2011) datapoints and of the Ota et al. (2010) LAE candidates at provides interesting constraints at high luminosities that cannot be fully captured in the (, ) parameter space of figure 5 and are better seen in figure 6. The main conclusion to be drawn from this test case is that to be consistent with our results, it does not require invoking a change in the Ly IGM transmission and therefore a change in the neutral-hydrogen fraction of the IGM.

In the second scenario, conversely, we consider a 60% change in luminosity from the LF of Ouchi et al. (2010). This scenario clearly requires, by construction, a significant quenching of the IGM Ly transmission. The present luminosity evolution scenario may be compatible with the claim by Fontana et al. (2010) of a significant quenching of Ly emission among LBGs at . Although strongly model dependent, a high neutral-hydrogen fraction of the IGM due to a still incomplete reionization would then be required in this scenario, e.g. according to the model of Santos (2004).

To conclude this discussion, we cannot safely decide on whether we are ‘seeing’ signatures of reionization in our results, as this depends on which assumptions we use for the LFs at lower redshifts. We call for clarifications of the LAE luminosity functions at redshift 6.5 and below, a prerequisite to drawing firmer conclusions on reionization as inferred from LAE observations. We also note that our results are inconsistent with the photometric samples from Tilvi et al. (2010) and Krug et al. (2011) at , but spectroscopic confirmation is required before drawing firmer conclusions. In the meantime, observations at continue. An ideal complement to the HAWK-I observations presented in this paper would be to probe higher luminosities (of the order of ergs.s) over significantly larger comoving volumes (a few times 10 Mpc).

Figure 6: Cumulative Ly luminosity functions. The blue (resp. red) lines show the cumulative luminosity functions at (resp. ) from Ouchi et al. (2008) (dotted-dashed lines), Hu et al. (2010) (plain lines) and Kashikawa et al. (2011) (dotted lines). The transitions to thin lines indicate the range of the luminosities probed by the observations. A fixed faint end slope of -1.5 is assumed. The parameter space probed by the HAWK-I observations is delimited by the black line, with the black dashed line corresponding to the HAWK-I observations and the null detection of the bright candidates of Hibon et al. (2010) mentioned in the text. The green dotted line corresponds to a 60% evolution in density of the Hu et al. (2010) LF, and the short-dashed-long-dashed line corresponds to a 60% evolution in luminosity of the Ouchi et al. (2010) LF. The red filled diamond corresponds to the Iye et al. (2006) LAE detection at a redshift of 6.96. The black open diamonds correspond to the Vanzella et al. (2011) LAE detections at redshifts of 7.008 and 7.109. The magenta downward-pointing filled (resp. open) triangles are the photometric candidates at from Tilvi et al. (2010) (resp. Krug et al. (2011)).

7 Conclusion

We searched for Ly emitters in three fields, two blank fields and one cluster field with the HAWK-I instrument at the VLT, using a NB filter centred at 1.06 m. Our data in this filter total 80 hrs of integration time. The total comoving volume is . We reached a 5 limiting magnitude that we use as detection threshold, of AB magnitude. We selected the objects from various colour criteria, which depend on the auxiliary data available for each field. We did not detect any object matching our selection criteria that would correspond to LAEs. We modelled the probability density function of high-z LAE populations including Poisson statistics and cosmic variance with a negative binomial distribution. From this statistical description and from the absence of LAE detections, we exclude a non-evolution scenario at (resp. 99%) confidence level from the LF of Hu et al. (2010) (resp. Ouchi et al. (2010); Kashikawa et al. (2011)). The large differences between the published LFs at prevent us from inferring a robust estimate of the Ly IGM transmission at , and therefore of the neutral-hydrogen fraction at this redshift. However, in all cases but the Hu et al. (2010) LF, a significant quenching of the Ly transmission by the IGM, probably due to reionization, is required.

Acknowledgements.
We acknowledge financial support from Agence Nationale de la Recherche (grant ANR-09-BLAN-0234-01). FC is supported by the Swiss National Science Foundation (SNSF). The Dark Cosmology Centre is funded by the Danish National Research Foundation.

References

  • Becker et al. (2001) Becker, R. H., Fan, X., White, R. L., et al. 2001, AJ, 122, 2850
  • Bertin & Arnouts (1996) Bertin, E. & Arnouts, S. 1996, A&AS, 117, 393
  • Bertin et al. (2002) Bertin, E., Mellier, Y., Radovich, M., et al. 2002, in Astronomical Society of the Pacific Conference Series, Vol. 281, Astronomical Data Analysis Software and Systems XI, ed. D. A. Bohlender, D. Durand, & T. H. Handley, 228–+
  • Bouwens et al. (2011) Bouwens, R. J., Illingworth, G. D., Oesch, P. A., et al. 2011, ApJ, 737, 90
  • Bradač et al. (2006) Bradač, M., Clowe, D., Gonzalez, A. H., et al. 2006, ApJ, 652, 937
  • Casertano et al. (2000) Casertano, S., de Mello, D., Dickinson, M., et al. 2000, AJ, 120, 2747
  • Cassata et al. (2011) Cassata, P., Le Fèvre, O., Garilli, B., et al. 2011, A&A, 525, A143+
  • Castellano et al. (2010) Castellano, M., Fontana, A., Boutsia, K., et al. 2010, A&A, 511, A20+
  • Clowe et al. (2006) Clowe, D., Bradač, M., Gonzalez, A. H., et al. 2006, ApJ, 648, L109
  • Dayal et al. (2011) Dayal, P., Maselli, A., & Ferrara, A. 2011, MNRAS, 410, 830
  • Dijkstra & Wyithe (2010) Dijkstra, M. & Wyithe, J. S. B. 2010, MNRAS, 408, 352
  • Fan et al. (2006) Fan, X., Carilli, C. L., & Keating, B. 2006, ARA&A, 44, 415
  • Finger et al. (2008) Finger, G., Dorn, R. J., Eschbaumer, S., et al. 2008, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 7021, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series
  • Fontana et al. (2010) Fontana, A., Vanzella, E., Pentericci, L., et al. 2010, ApJ, 725, L205
  • Gehrels (1986) Gehrels, N. 1986, ApJ, 303, 336
  • Gonzalez et al. (2009) Gonzalez, A. H., Clowe, D., Bradač, M., et al. 2009, ApJ, 691, 525
  • Grazian et al. (2006) Grazian, A., Fontana, A., de Santis, C., et al. 2006, A&A, 449, 951
  • Hibon et al. (2010) Hibon, P., Cuby, J., Willis, J., et al. 2010, A&A, 515, A97+
  • Hu et al. (2010) Hu, E. M., Cowie, L. L., Barger, A. J., et al. 2010, ApJ, 725, 394
  • Hu et al. (2002) Hu, E. M., Cowie, L. L., McMahon, R. G., et al. 2002, ApJ, 568, L75
  • Iye et al. (2006) Iye, M., Ota, K., Kashikawa, N., et al. 2006, Nature, 443, 186
  • Kashikawa et al. (2006) Kashikawa, N., Shimasaku, K., Malkan, M. A., et al. 2006, ApJ, 648, 7
  • Kashikawa et al. (2011) Kashikawa, N., Shimasaku, K., Matsuda, Y., et al. 2011, ApJ, 734, 119
  • Kobayashi et al. (2007) Kobayashi, M. A. R., Totani, T., & Nagashima, M. 2007, ApJ, 670, 919
  • Kobayashi et al. (2010) Kobayashi, M. A. R., Totani, T., & Nagashima, M. 2010, ApJ, 708, 1119
  • Krug et al. (2011) Krug, H., Veilleux, S., Tilvi, V., et al. 2011, arXiv:1106.6055
  • Larson et al. (2011) Larson, D., Dunkley, J., Hinshaw, G., et al. 2011, ApJS, 192, 16
  • Laursen et al. (2011) Laursen, P., Sommer-Larsen, J., & Razoumov, A. O. 2011, ApJ, 728, 52
  • Maizy et al. (2010) Maizy, A., Richard, J., de Leo, M. A., Pelló, R., & Kneib, J. P. 2010, A&A, 509, A105+
  • Malhotra & Rhoads (2004) Malhotra, S. & Rhoads, J. E. 2004, ApJ, 617, L5
  • Marigo et al. (2008) Marigo, P., Girardi, L., Bressan, A., et al. 2008, A&A, 482, 883
  • Nakamura et al. (2011) Nakamura, E., Inoue, A. K., Hayashino, T., et al. 2011, MNRAS, 55
  • Nilsson et al. (2007) Nilsson, K. K., Orsi, A., Lacey, C. G., Baugh, C. M., & Thommes, E. 2007, A&A, 474, 385
  • Ota et al. (2010) Ota, K., Iye, M., Kashikawa, N., et al. 2010, ApJ, 722, 803
  • Ouchi et al. (2008) Ouchi, M., Shimasaku, K., Akiyama, M., et al. 2008, ApJS, 176, 301
  • Ouchi et al. (2010) Ouchi, M., Shimasaku, K., Furusawa, H., et al. 2010, ApJ, 723, 869
  • Pickles (1998) Pickles, A. J. 1998, PASP, 110, 863
  • Retzlaff et al. (2010) Retzlaff, J., Rosati, P., Dickinson, M., et al. 2010, A&A, 511, A50+
  • Richard et al. (2011) Richard, J., Kneib, J.-P., Ebeling, H., et al. 2011, MNRAS, L245+
  • Richard et al. (2008) Richard, J., Stark, D. P., Ellis, R. S., et al. 2008, ApJ, 685, 705
  • Santos (2004) Santos, M. R. 2004, MNRAS, 349, 1137
  • Schechter (1976) Schechter, P. 1976, ApJ, 203, 297
  • Somerville et al. (2004) Somerville, R. S., Lee, K., Ferguson, H. C., et al. 2004, ApJ, 600, L171
  • Stark et al. (2011) Stark, D. P., Ellis, R. S., & Ouchi, M. 2011, ApJ, 728, L2+
  • Taniguchi et al. (2005) Taniguchi, Y., Ajiki, M., Nagao, T., et al. 2005, PASJ, 57, 165
  • Tanvir et al. (2009) Tanvir, N. R., Fox, D. B., Levan, A. J., et al. 2009, Nature, 461, 1254
  • Tilvi et al. (2010) Tilvi, V., Rhoads, J. E., Hibon, P., et al. 2010, ApJ, 721, 1853
  • Trenti & Stiavelli (2008) Trenti, M. & Stiavelli, M. 2008, ApJ, 676, 767
  • Vanzella et al. (2011) Vanzella, E., Pentericci, L., Fontana, A., et al. 2011, ApJ, 730, L35+
  • Willis & Courbin (2005) Willis, J. P. & Courbin, F. 2005, MNRAS, 357, 1348
  • Willis et al. (2008) Willis, J. P., Courbin, F., Kneib, J.-P., & Minniti, D. 2008, MNRAS, 384, 1039
  • Willott et al. (2010) Willott, C. J., Delorme, P., Reylé, C., et al. 2010, AJ, 139, 906
  • Yang & Saslaw (2011) Yang, A. & Saslaw, W. C. 2011, ApJ, 729, 123
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
117338
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description