Multispectral video acquisition

Introduction

Light emitted from illumination sources or reflected from scene objects generally spans a broad range of wavelengths. Though trichromatic sensing suffices for the human visual system in many circumstances, multispectral imaging with more than three spectral measurements per scene point provides greater information about the captured scene and objects. This information can be used to the advantage of a number of machine vision applications, including illumination analysis, object tracking, and material recognition.

Highlights

A Prism-Mask System for Multispectral Video Acquisition

This paper presents a prism-mask system for capturing multispectral videos. The system is composed of a triangular prism, a monochrome camera, and an occlusion mask. Incoming light beams from the scene are sampled by the occlusion mask, dispersed into their constituent spectra by the triangular prism, and then captured by the monochrome camera. Our system is capable of capturing frames with high spectral resolution at video rates. It also allows for different trade-offs between spectral and spatial resolution by adjusting the focal length of the camera. We demonstrate multispectral video acquisition with various spectral resolutions and spatial resolutions, as well as different frame rates. The effectiveness of our system is further evaluated with several applications, including human skin detection, physical material recognition, video segmentation, RGB video generation, and illumination identification.

Cao, X., Du, H., Tong, X., Dai, Q., & Lin, S. (2011). A prism-mask system for multispectral video acquisition. IEEE transactions on pattern analysis and machine intelligence, 33(12), 2423-2435.

Content-adaptive high-resolution hyperspectral video acquisition with a hybrid camera system

We present a hybrid camera system that combines optical designs with computational processing to achieve content-adaptive high-resolution hyperspectral video acquisition. In particular, we record two video streams: one high-spatial resolution RGB video and one low-spatial resolution hyperspectral video in which the recorded points are dynamically selected using a spatial light modulator (SLM). Then through video-frame registration and a spatio-temporal spreading of the co-located spectral/RGB information, video with high spatial and spectral resolution is produced. The sampling patterns on the SLM are generated on-the-fly according to the scene content, which fully exploits the self-adaptivity of the hybrid camera system. With an experimental prototype, we demonstrate significantly improved accuracy and efficiency as compared to the state-of-the-art.

Ma, C., Cao, X., Wu, R., & Dai, Q. (2014). Content-adaptive high-resolution hyperspectral video acquisition with a hybrid camera system Optics letters, 39(4), 937-940.

Bispectral coding: compressive and high-quality acquisition of fluorescence and reflectance

Fluorescence widely coexists with reflectance in the real world, and an accurate representation of these two components in a scene is vitally important. Despite the rich knowledge of fluorescence mechanisms and behaviors, traditional fluorescence imaging approaches are quite limited in efficiency and quality. To address these two shortcomings, we propose a bispectral coding scheme to capture fluorescence and reflectance: multiplexing code is applied to excitation spectrums to raise the signal-to-noise ratio, and compressive sampling code is applied to emission spectrums for high efficiency. For computational reconstruction from the sparse coded measurements, the redundancy in both components promises recovery from sparse measurements, and the difference between their redundancies promises accurate separation. Mathematically, we cast the reconstruction as a joint optimization, whose solution can be derived by the Augmented Lagrange Method. In our experiment, results on both synthetic data and real data captured by our prototype validate the proposed approach, and we also demonstrate its advantages in two computer vision tasks—photorealistic relighting and segmentation.

Suo, J., Bian, L., Chen, F., & Dai, Q. (2014). Bispectral coding: compressive and high-quality acquisition of fluorescence and reflectance. Optics express, 22(2), 1697-1712.