In-Pixel Processing and In-Memory Computing in CMOS Image Sensors for Augmented Reality

PASSCODE: 1005

Vision-based tracking systems in emerging augmented reality (AR) devices provide a spatial understanding of user’s gestures and surrounding environments, enabling a fusion of computer-generated objects with real-world spaces. Yet, stringent requirements on the tracking system’s energy dissipation, latency, or dynamic range (DR), remain as major challenges for realistic AR experience. In this thesis, two CMOS image sensors (CISs) are presented focusing on pixel-level circuits and signal processing that improve the system performance.

The first part demonstrates a CIS embedded with in-memory computing of machine-learning face detection classifiers. The proposed classifiers are directly computed from a global shutter pixel array simultaneously with an image readout, which reduces the system latency. The classifiers only consume a latency of 5.1 ms with an energy efficiency of 8.2TOPS/W, improving the system energy-latency product by 2.3×.

The second part demonstrates a high DR CIS with pixel-level temporal oversampling for AR operation in outdoor environments. The intrascene DR is extended by capturing two images with long and short exposures, which are then merged. The short exposure image is temporally oversampled with pixel-level sigma-delta modulator and extends DR by 56 dB, while suppressing a signal-to-noise ratio dip under 10 dB. Aligned short/long exposures with partial charge transfer allows motion artifact/LED flicker-free HDR imaging.

CHAIR: Professor Euisik Yoon