Widefield lensless endoscopy with a multicore fiber
We demonstrate pixelation-free real-time widefield endoscopic imaging through an aperiodic multicore fiber (MCF) without any distal opto-mechanical elements or proximal scanners. Exploiting the memory effect in MCFs the images in our system are directly obtained without any post-processing using a static wavefront correction obtained from a single calibration procedure. Our approach allows for video-rate 3D widefield imaging of incoherently illuminated objects with imaging speed not limited by the wavefront shaping device refresh rate.
Fiber-optic microendoscopes allow for minimally invasive high-resolution imaging deep within living organisms. Over the last decades they continue to gain in versatility with the miniaturization of fiber-based devices and multimodal imaging capabilities. A new class of such devices, fiber-based lensless endoscopes, operating without any distal optical or mechanical elements, enabled extreme miniaturization of the probe dimensions down to a few hundred micrometers, permitting minimally invasive imaging Čižmár and Dholakia (2012); Choi et al. (2012); Andresen et al. (2013a); Sivankutty et al. (2016a); Papadopoulos et al. (2012); Kim et al. (2014). Image formation in the lensless endoscopes, capable of producing focal planes at various distances from the fiber tip, relies on either raster scanning or widefield modalities. In order to reach real-time image acquisition rates, these systems require in the former case ultrafast devices capable of wavefront shaping (typically deformable mirrors) or beam scanning Andresen et al. (2013b), or real-time computation in the latter Choi et al. (2012); Porat et al. (2016).
Here we show that, using a slow wavefront-shaping device (spatial light modulator, SLM) and an MCF with weakly coupled cores, it is straightforward to achieve conventional widefield imaging in real time using a single calibration procedure. Building on the framework developed for imaging through scattering media Katz et al. (2012) and relying on the practically infinite optical memory effect in such MCFs Sivankutty et al. (2016b); Stasio et al. (2015), we demonstrate widefield imaging of incoherently illuminated objects.
A conceptual illustration for our technique with the corresponding numerical simulation is depicted in Figure 1. The schematic in Fig. 1a represents a conventional widefield microscope where light, scattered from an incoherently illuminated object, gets collected by a 4f system, thus forming an image on the other side of it. For such an imaging system with a lateral intensity point spread function , image intensity distribution is related to the object through a convolution operation: .
Typically, an MCF acts as an imaging conduit, transmitting object information from one fiber endface to another. In earlier implementations, the individual cores sample the object directly Gmitro and Aziz (1993); Klimas and Entcheva (2014), giving rise to two important restrictions : i) pixelation due to inter-core separation and ii) imaging restricted to fiber facet itself, putting the probe in contact with the sample. The ability to operate at flexible working distances is highly desirable in context of endoscopic applications. In our previous works Andresen et al. (2013b, a), we reported MCF devices meeting both of the mentioned requirements. The combination of a wavefront shaping device and an MCF can effectively function in a manner analogous to the common 4f system. A considerable difference being its PSF, which exhibits significant side lobes. They arise from the fact that the pupil of an MCF is segmented and such discontinuities give rise to prominent side lobes Hyvärinen et al. (2012). This in turn affects the image, transmitted through the system, as illustrated in Fig. 1b, resulting in ghost images of the object. Recently we showed Sivankutty et al. (2016b) that the randomness in the MCF core positions can lead to a considerable reduction of the side lobes intensity in the PSF of such an imaging system. A simulation of an imaging experiment with such an aperiodic MCF is shown in Fig. 1b, where the PSF is calculated given the cores distribution in the real fiber used throughout this work. While the images exhibit replicas of the object, their intensity is at least 2.5 times lower compared to the central image. Commercial fiber bundles do exhibit variations in core geometry and spacing in view of decreasing the inter-core coupling Chen et al. (2008), hence we expect a larger reduction in the side lobe intensity Stasio et al. (2015).
Implementations of the MCF with wavefront shaping, reported so far, have all employed raster scanning for imaging Thompson et al. (2011); Andresen et al. (2013b, a); Stasio et al. (2015). Unlike in endoscopes based on multimode fibers (MMF), this becomes trivial in MCFs due to their very large memory effect Sivankutty et al. (2016b). As there is little or no cross-talk between fiber cores, the transmission matrix of an MCF is practically diagonal. Hence, any phase gradient at the proximal end is preserved during light propagation through the fiber. This phenomenon has been employed to remotely scan the beam by applying a global tip-tilt on the composite wavefront entering the MCF. Since these are relatively simple patterns, the SLM can now be decoupled from having to perform the scanning and conventional galvanometer-based scanners were employed allowing imaging rates of 11 fps Andresen et al. (2013b).
In this Letter, we propose and demonstrate the extension of the memory effect of MCFs to the widefield imaging framework. The reasoning is relatively straightforward, since a translation of a point source in the object plane would result in a phase gradient which is preserved through the fiber (linear in case of the transverse plane and quadratic for the axial plane). This results in an accurate mapping of any object shift to the image. These concepts are reminiscent of earlier experiments in scattering media Vellekoop and Aegerter (2010); Katz et al. (2012), and the additional key advantage in the case of the MCF being that the memory effect does not play a limiting role in the imaging process.
We focus on the experimental evaluation of such widefield incoherent imaging employing the memory effect to enhance speed and simplicity. The novel design and the original fabrication approach of this fiber were previously reported in Sivankutty et al. (2016b). Figure 2c depicts the fiber fabricated with the following parameters: individual core diameter m, its numerical aperture , master triangular lattice pitch m, and the randomness parameter PR (see Sivankutty et al. (2016b) for details). The outer diameter of the probe is about 360 m (Fig. 2c) and the length of the fiber used in this experiment was 40 cm, ensuring no considerable inter-core group delay dispersion Andresen et al. (2015).
A simplified scheme of the experimental set-up, used for the widefield imaging with this MCF, is shown in Figs. 2a,b. The laser beam from a femtosecond source (Amplitude Systèmes t-Pulse, nm, 180 fs, 50 MHz) is extended with a telescope (L1, L2) to overfill the aperture of the 2D liquid crystal SLM (Hamamatsu X10468-07). The latter is used to shape the segmented wavefront entering the MCF via its proximal endface. During the initial system calibration step, a microlens segment inscribed on the SLM for each individual fiber core produces a focal spot at the focal length from the SLM face, and is spatially scanned around its initial position in order to optimize the coupling into the corresponding core. The array of the optimized focal spots is then imaged onto the MCF proximal endface via a system of lenses (L3, MO1). A focal plane IP is created at a distance m away from the MCF distal end and imaged via another telescope system (MO2, L4) onto a camera (CCD1) for calibration and testing purposes. Output polarization from different cores of such non polarization maintaining MCF is arbitrary Sivankutty et al. (2016c), therefore we employ a linear polarizer (P, Thorlabs LPNIR100) to discard any concomitant effects. After the initial system calibration and compensation of the distal wavefront for the intrinsic MCF phase distortion, one obtains a characteristic PSF (Fig. 2d) comprising a central spot surrounded by six dimmer replicas distributed on a circumference with m. In the linear imaging regime, the brightest speckle contrast relative to the central peak () is 0.4, measured in the field of view (FoV) center (Fig. 2d). We verify the PSF variation across the FoV, in Fig. 2e we summarize such measurements for off-axial points, showing the variation of PSF FWHM less than 1 m and on the FoV edge, which should not drastically decrease the imaging performance of the widefield technique.
Using this fiber, we perform a series of proof-of-concept experiments, described in the following. The experimental set-up, used for the fiber calibration (Fig. 2a), features two conjugate focal planes: OP and IP (where a particular distance for the IP can be flexibly chosen during the calibration step). Unlike the calibration step, where it is required to have a spatially and temporally coherent source, the following imaging experiments are performed with spatially incoherent illumination of the object (nevertheless, the bandwidth of the illumination source has to be smaller than the speckle spectral correlation width). We now perform an incoherent projection of an amplitude mask from the OP to the IP plane; we denote such operation as forward projection. The related experiment is performed with a reflective object from United States Air Force (USAF) resolution chart (Fig. 3a). The phase mask on the 2D-SLM is the same used to correct for the intrinsic MCF phase distortion and does not change throughout the experiment, unless one wishes to switch the projection to another working plane (different ). A number ’5’ target (object height m) was placed in the OP and illuminated incoherently by placing a rotating diffuser (D in Fig. 2a) between the object plane and the laser source. The position of the diffuser along the beam propagation direction is chosen to create in the OP a sufficiently large illumination area, slightly exceeding the dimensions of the used mask. The measured projection in the IP is shown in Fig. 3b; it is clear that the set-up performs like a conventional 4f imaging system with only a single calibration aided by the memory effect. As the SLM does not need to be updated any further, we can easily perform high speed image acquisition (exposure time for the presented example was 30 ms). For a real-time imaging experiment of a moving target, see Visualization 1.
Next, we perform a widefield imaging experiment in the epi direction, using the modified set-up shown in Fig. 2b. In this configuration the spatially incoherent illumination (the same as for the forward projection experiment) reaches the sample plane (OP) after passing the MO2, and the IP is matched directly to the CCD2 camera plane (note that the physical locations of IP and OP are switched as compared to the forward projection experiment). A linear polarizer (P) is used after the MCF proximal end in the same scope as in the forward projection set-up, as we measure only the scalar transmission matrix Tripathi et al. (2012). We use two types of USAF targets – positive and negative – to compare the operation of our imaging system in different sample configurations, particularly in terms of the influence of the side lobes. In the first case we use a positive target consisting of number ’6’ (Fig. 3c); object height is m, which is almost 3 times less compared to the distance between the side lobes of the system PSF. This results in an image on relatively homogeneous background (Fig. 3d) without any overlaying ghost replicas. In the case of negative target (Fig. 3e), given the object height m and the distance between replicas, the object convolution with the system PSF results in ghost images of the object that begin to overlap with the central (brightest) ’2’. Unlike in TPEF microscopy Sivankutty et al. (2016b) where this weak background is screened by the inherent nonlinearity and does not contribute to image formation, these now result in ghost images albeit with reduced intensity. As it can be seen from Fig. 3f, the image associated with the central lobe is still the brightest one and can be easily distinguished from the background generated by the side lobes and any weak speckle.
We note that earlier results from our group on the same MCF achieved a FoV of 120 m (0.064), whereas in the present work the illumination area was restricted to cover a FoV of 50 m (0.03) due to lower experimental SNR.
Another advantage of the direct imaging approach with an SLM over speckle-correlation based techniques Porat et al. (2016) for widefield imaging is that our approach offers a degree of optical sectioning due to its 3D transfer function Streibl (1985).
We performed an experimental measurement of the depth of field (DoF) in the following manner. After distortion compensation, the calculated phase pattern is displayed on the SLM in order to obtain a focal plane at m from the MCF distal end. Next, a diffraction-limited point source is placed at different and the DoF is evaluated from the stack of images measured at CCD2, cf. Fig. 2b. This results in a Gaussian distribution (Fig. 4f) with FWHM m, which is in qualitative agreement with the expected DoF (32 m). Then, using the same experimental layout as in Fig. 2b and the same test object as in Fig. 3e, placed into the OP at m away from the MCF distal tip, we show in and out of focus images of the target (Fig. 4a) by switching the focal planes through displaying on the SLM the respective differential phase patterns with no mechanical translation of the sample or the fiber. As expected, the object appears in focus and then is defocused on the CCD2 plane (IP). For the full stack of 12 imaged focal planes between and 700 m, see Visualization 2.
Considering the measured , we further demonstrate proof of concept optical sectioning experiments (Fig. 4a) with a 3D phantom, schematically depicted in Fig. 4e. It consists of 50 nm thick layer of gold flakes, deposited onto a 130 m thick glass cover slip (#1). From the bottom side of the latter, we stack a positive USAF target (Thorlabs R1DS1P) in a manner that the separation between two metal layers is equal to the cover slip thickness. Therefore, when introduced into widefield imaging set-up, the gold flakes layer is situated at 1 m away from the MCF distal end, whereas the USAF target layer is at 2 m (Fig. 4e). By adjusting the phase mask on the SLM in order to translate the endoscope focal plane along axis, we could acquire clear images at several without moving either the fiber probe or the sample. Images of the two planes of interest – with the gold flake (Fig. 4b) and with USAF target group 7 elements 3–5 (Fig. 4d), plus an intermediate plane at m (Fig. 4c) are shown with the overlaying white dashed lines indicating the outline and the position of the gold flake. Remarkably, we are able to focus on both these planes remotely using the SLM, demonstrating pixelation-free and 3D-resolved imaging. For the full stack of 8 focal planes, imaged between and 730 m, see Visualization 3. We believe this is the first ever demonstration of widefield 3D-resolved imaging at multiple depths in lensless endoscopy employing linear contrast and we expect this to be significantly valuable in the context of imaging 3D structures. In the current implementation, the presented widefield lensless endoscope is not resilient to fiber bending which would change the phase dispersion within the MCF. However, solutions under development may eventually permit real-time compensation of these effects, see e.g. Andresen et al. (ress) and references therein.
We have demonstrated real-time pixelation-free widefield imaging through an aperiodic MCF using the optical memory effect without any distal opto-mechanical elements and requiring no further post-processing. Both the forward projection of an amplitude mask from the proximal to distal side of the endoscopic system and epi widefield imaging were shown. By employing wavefront control at the fiber proximal end, we showed that our system is capable of optical sectioning – producing clear images of different focal planes within a quasi-3D sample, and it does so without the need to physically displace either the MCF or the sample.
In the future, we expect that widefield imaging could be combined with nonlinear imaging, e.g. TPEF imaging in a lensless endoscope Andresen et al. (2013b). Fluorescence sources or another broadband illumination whose bandwidth is narrower than the MCF’s speckle spectral correlation bandwidth can be also used without significantly affecting the imaging performance (see Porat et al. (2016) for details). Considering the robustness, fabrication simplicity of the presented aperiodic MCF with low inter-core coupling, as well as facilitated ultrashort pulse delivery and the small effective probe diameter (an order of magnitude smaller compared to the existing scanning endoscope solutions), such a fiber emerges as a promising candidate in the scope of miniaturized imaging systems.
Agence Nationale de la Recherche (ANR) (ANR-11-INSB-0006, ANR-10-INSB-04-01, ANR-14-CE17-0004-01); Aix-Marseille Université (ANR-11-IDEX-0001-02); Université Lille 1 (ANR-11-LABX-0007, ANR-11-EQPX-0017, CPER P4S Région Nord Pas-de-Calais); Institut National de la Santé et de la Recherche Médicale (Inserm) (PC201508); SATT Sud-Est GDC Lensless endoscope; European research council grant no. 677909; The Azrieli foundation; CNRS/Weizmann NaBi European Associated Laboratory.
Authors thank Juan de Torres for providing the gold flakes sample.
- Čižmár and Dholakia (2012) T. Čižmár and K. Dholakia, Nature Communications 3, 1027 (2012).
- Choi et al. (2012) Y. Choi, C. Yoon, M. Kim, T. D. Yang, C. Fang-Yen, R. R. Dasari, K. J. Lee, and W. Choi, Physical Review Letters 109, 203901 (2012).
- Andresen et al. (2013a) E. R. Andresen, G. Bouwmans, S. Monneret, and H. Rigneault, Optics express 21, 20713 (2013a).
- Sivankutty et al. (2016a) S. Sivankutty, E. R. Andresen, R. Cossart, G. Bouwmans, S. Monneret, and H. Rigneault, Optics Express 24, 825 (2016a).
- Papadopoulos et al. (2012) I. N. Papadopoulos, S. Farahi, C. Moser, and D. Psaltis, Optics Express 20, 10583 (2012).
- Kim et al. (2014) D. Kim, J. Moon, M. Kim, T. D. Yang, J. Kim, E. Chung, and W. Choi, Optics Letters 39, 1921 (2014).
- Andresen et al. (2013b) E. R. Andresen, G. Bouwmans, S. Monneret, and H. Rigneault, Optics letters 38, 609 (2013b).
- Porat et al. (2016) A. Porat, E. R. Andresen, H. Rigneault, D. Oron, S. Gigan, and O. Katz, Optics Express 24, 16835 (2016), arXiv:1601.01518 .
- Katz et al. (2012) O. Katz, E. Small, and Y. Silberberg, Nature Photonics 6, 549 (2012).
- Sivankutty et al. (2016b) S. Sivankutty, V. Tsvirkun, G. Bouwmans, D. Kogan, D. Oron, E. R. Andresen, and H. Rigneault, Optics Letters 41, 3531 (2016b), arXiv:1606.08169 .
- Stasio et al. (2015) N. Stasio, D. B. Conkey, C. Moser, and D. Psaltis, Optics Express 23, 30532 (2015).
- Gmitro and Aziz (1993) A. F. Gmitro and D. Aziz, Optics Letters 18, 565 (1993).
- Klimas and Entcheva (2014) A. Klimas and E. Entcheva, Journal of Biomedical Optics 19, 080701 (2014).
- Hyvärinen et al. (2012) H. J. Hyvärinen, S. Rehman, J. Tervo, J. Turunen, and C. J. R. Sheppard, Optics Letters 37, 903 (2012).
- Chen et al. (2008) X. Chen, K. L. Reichenbach, and C. Xu, Optics Express 16, 21598 (2008).
- Thompson et al. (2011) A. J. Thompson, C. Paterson, M. A. A. Neil, C. Dunsby, and P. M. W. French, Optics Letters 36, 1707 (2011).
- Vellekoop and Aegerter (2010) I. M. Vellekoop and C. M. Aegerter, Optics Letters 35, 1245 (2010).
- Andresen et al. (2015) E. R. Andresen, S. Sivankutty, G. Bouwmans, L. Gallais, S. Monneret, and H. Rigneault, Journal of the Optical Society of America B 32, 1221 (2015).
- Sivankutty et al. (2016c) S. Sivankutty, E. R. Andresen, G. Bouwmans, T. G. Brown, M. A. Alonso, and H. Rigneault, Optics Letters 41, 2105 (2016c).
- Tripathi et al. (2012) S. Tripathi, R. Paxman, T. Bifano, and K. C. Toussaint, Optics Express 20, 16067 (2012).
- Streibl (1985) N. Streibl, Journal of the Optical Society of America A 2, 121 (1985).
- Andresen et al. (ress) E. R. Andresen, S. Sivankutty, V. Tsvirkun, G. Bouwmans, and H. Rigneault, Journal of Biomedical Optics (In press).