De-scattering with Excitation Patterning (DEEP) Enables Rapid Wide-field Imaging Through Scattering Media

Very interesting stuff from the people at MIT regarding imaging through scattering media. Recently, multiple approaches taking advantage of temporal focusing (TF) increased efficiency inside scattering media when using two-photon microscopy have been published, and this goes a step further.

Here, the authors use wide-field structured illumination, in combination with TF, to obtain images with a large field-of-view and a slow number of camera acquisitions. To do so, they sequentially project a set of random structured patterns using a digital micromirror device (DMD). Using the pictures acquired for each illumination pattern in combination with the point-spread-function (PSF) of the imaging system allows to recover images of different biological samples without the typical scattering blur.

Optical design and working principle of the system. Figure extracted from “De-scattering with Excitation Patterning (DEEP) Enables Rapid Wide-field Imaging Through Scattering Media,” Dushan N. Wadduwage et al., at https://arxiv.org/abs/1902.10737

De-scattering with Excitation Patterning (DEEP) Enables Rapid Wide-field Imaging Through Scattering Media

by Dushan N. Wadduwage et al., at arXiv.

Abstract:

From multi-photon imaging penetrating millimeters deep through scattering biological tissue, to super-resolution imaging conquering the diffraction limit, optical imaging techniques have greatly advanced in recent years. Notwithstanding, a key unmet challenge in all these imaging techniques is to perform rapid wide-field imaging through a turbid medium. Strategies such as active wave-front correction and multi-photon excitation, both used for deep tissue imaging; or wide-field total-internal-refection illumination, used for super-resolution imaging; can generate arbitrary excitation patterns over a large field-of-view through or under turbid media. In these cases, throughput advantage gained by wide-field excitation is lost due to the use of point detection. To address this challenge, here we introduce a novel technique called De-scattering with Excitation Patterning, or ‘DEEP’, which uses patterned excitation followed by wide-field detection with computational imaging. We use two-photon temporal focusing (TFM) to demonstrate our approach at multiple scattering lengths deep in tissue. Our results suggest that millions of point-scanning measurements could be substituted with tens to hundreds of DEEP measurements with no compromise in image quality.

Rapid broadband characterization of scattering medium using hyperspectral imaging

People at LKB (and St. Andrews) keep shining light into scattering media. This time, they have developed a cool approach for measuring the multispectral Transmission Matrix (MSTM) of a medium. This knowledge allows to control each spectral component of a light beam when travelling through the medium, which permits to shape, for example, the spectral and temporal profiles of light pulses. This is quite nice, as can be used to generate tight focci inside biological tissues, improving the performance of nonlinear microscopy techniques.

Usually, the measurement of the MSTM entails a long iterative process (basically you just measure the TM for each spectral channel you want to characterize). This is not always possible (usually you do not have a laser with all the wavelengths you need to measure), and also tends to be slow (which is a problem if you want to measure the MSTM of a changing medium). Here the authors tackle this problem by performing a wavelength-to-spatial mapping, thus measuring the spatio-spectral information in just one shot of a CCD camera. To do so, they use a clever design with a lenslet array and a dispersion grating. In this way, the total time it takes to acquire the MSTM is reduced in ~2 orders of magnitude. Elegant, simple, and fast.

Design concept for the spectral measurements using a lenslet array and a single CCD sensor. Extracted from “Rapid broadband characterization of scattering medium using hyperspectral imaging,” A. Boniface et al., https://www.osapublishing.org/optica/abstract.cfm?uri=optica-6-3-274

Rapid broadband characterization of scattering medium using hyperspectral imaging

by Antoine Boniface et al., at Optica

Abstract:

Scattering of a coherent ultrashort pulse of light by a disordered medium results in a complex spatiotemporal speckle pattern. The form of the pattern can be described by knowledge of a spectrally dependent transmission matrix, which can in turn be used to shape the propagation of the pulse through the medium. We introduce a method for rapid measurement of this matrix for the entire spectrum of the pulse based on a hyperspectral imaging system that is close to 2 orders of magnitude faster than any approach previously reported. We demonstrate narrowband as well as spatiotemporal refocusing of a femtosecond pulse temporally stretched to several picoseconds after propagation through a multiply scattering medium. This enables new routes for multiphoton imaging and manipulation through complex media.

Light transport and imaging through complex media & Photonics West 2018

Last ~20 days have been completely crazy. First, I went to a meeting organized by the Royal Society: Light transport and imaging through complex media. It was amazing. Beautiful place, incredible researchers, and a nice combination of signal processing and optical imaging. I am sure I will be looking for future editions.

After that, I assisted Photonics West. Both BIOS and OPTO were full of interesting talks. Scattering media, adaptive optics, DMDs, some compressive sensing… Fantastic week. There I talked about two recent works we made in Spain: balanced photodetection single-pixel imaging and phase imaging using a DMD and a lateral position detector. Both contributions were very well received, and I am happy with the feedback I got. So many new ideas… now I need some time to implement them! I plan on writing a bit here on the blog about the last work, which has been published in the last issue of Optica.

 

Some of the cool stuff I heard about:

Valentina Emiliani – Optical manipulation of neuronal circuits by optical wave front shaping. Very cool implementations combining multiple SLMs and temporal focusing to see how neurons work.

Richard Baraniuk – Phase retrieval: tradeoffs and a new algorithm. How to recover phase information from intensity measurements. Compressive sensing and inverse problems. Very interesting, and a really good speaker. It is difficult to find someone capable of explaining these concepts as easily as Richard.

Michael Unser – GlobalBioIm

When being confronted with a new imaging problem, the common experience is that one has to reimplement (if not reinvent) the wheel (=forward model + optimization algorithm), which is very time consuming and also acts as a deterrent for engaging in new developments. This Matlab library aims at simplifying this process by decomposing the workflow onto smaller modules, including many reusable ones since several aspects such as regularization and the injection of prior knowledge are rather generic. It also capitalizes on the strong commonalities between the various image formation models that can be exploited to obtain fast, streamlined implementations.

Oliver Pust – High spatial resolution hyperspectral camera based on a continously variable filter. Really cool concept of merging a continous filter and multiple expositions to obtain hyperspectral information and even 3D images.

Seungwoo Shin – Exploiting a digital micromirror device for a multimodal approach combinning optical diffraction tomography and 3D structured illumination microscopy. I am always happy to see cool implementations with DMDs. This is one of them. KAIST delivers.

We propose a multimodal system combining ODT and 3-D SIM to measure both 3-D RI and fluorescence distributions of samples with advantages including high spatiotemporal resolution as well as molecular specificity. By exploiting active illumination control of a digital micromirror device and two different illumination wavelengths, our setup allows to individually operate either ODT or 3-D SIM. To demonstrate the feasibility of our method, 3-D RI and fluorescence distributions of a planar cluster of fluorescent beads were reconstructed. To further demonstrate the applicability, a 3-D fluorescence and time-lapse 3-D RI distributions of fluorescent beads inside a HeLa cell were measured.

Post featured image extracted from here.

Focusing light through dynamical samples using fast continuous wavefront optimization

The guys at LKB keep going inside turbid media. This time, they have done it really fast. By using a phase spatial light modulator and with the help of a FPGA card, they were able to focus light through a scattering medium at a rate of ~4 kHz.

This is trying to solve a common problem in biological systems when you use the Transmission Matrix approach: live systems evolve, and thus the matrix that you measure is not valid after a really short time.

For me, this is a really nice technical implementation (and not an easy one to do) merging electronics, computer science, and optics to tackle a well defined biological problem.

Focusing light through dynamical samples using fast continuous wavefront optimization,

B. Blochet et al, at Optics Letters

(featured image extracted from Fig. 1 of the manuscript)

Abstract:

We describe a fast continuous optimization wavefront shaping system able to focus light through dynamic scattering media. A micro-electro-mechanical system-based spatial light modulator, a fast photodetector, and field programmable gate array electronics are combined to implement a continuous optimization of a wavefront with a single-mode optimization rate of 4.1 kHz. The system performances are demonstrated by focusing light through colloidal solutions of TiO2 particles in glycerol with tunable temporal stability.

 

Imaging through glass diffusers using densely connected convolutional networks

I just found a new paper by the group of G. Barbastathis at MIT.

Imaging through glass diffusers using densely connected convolutional networks,

S. Li et al, Submitted on 18 Nov 2017, https://arxiv.org/abs/1711.06810

(featured image from Fig. 3 of the manuscript)

Abstract,

Computational imaging through scatter generally is accomplished by first characterizing the scattering medium so that its forward operator is obtained; and then imposing additional priors in the form of regularizers on the reconstruction functional so as to improve the condition of the originally ill-posed inverse problem. In the functional, the forward operator and regularizer must be entered explicitly or parametrically (e.g. scattering matrices and dictionaries, respectively.) However, the process of determining these representations is often incomplete, prone to errors, or infeasible. Recently, deep learning architectures have been proposed to instead learn both the forward operator and regularizer through examples. Here, we propose for the first time, to our knowledge, a convolutional neural network architecture called “IDiffNet” for the problem of imaging through diffuse media and demonstrate that IDiffNet has superior generalization capability through extensive tests with well-calibrated diffusers. We found that the Negative Pearson Correlation Coefficient loss function for training is more appropriate for spatially sparse objects and strong scattering conditions. Our results show that the convolutional architecture is robust to the choice of prior, as demonstrated by the use of multiple training and testing object databases, and capable of achieving higher space-bandwidth product reconstructions than previously reported.

Basically they have trained a neural network to ‘solve’ the path of light traveling through a scattering medium, thus being able to recover images hidden by glass diffusers. It may sound simple, but thousands of scientists are trying to see objects hidden by scattering media. We are seeing the first steps of the combination between neural networks, machine learning, and optics to go beyond physical constraints imposed by nature (see inside our bodies with visible light, see through fog, etc.).

Fig8_Barbastatis
Fig. 7 of the paper, with some nice results.