Machine Learning and Event-Based Camera for Microlens Array-Based Optical Systems

Date of Award

5-9-2026

Degree Name

Ph.D. in Electrical Engineering

Department

Department of Electrical and Computer Engineering

Advisor/Chair

Keigo Hirakawa

Abstract

Event-based cameras are optimal for applications that require the sensor to detect fast, sparse signals, especially when integrated into a larger system that must respond to the signal in real-time. In these scenarios, it is imperative that the event-based algorithm is both highly performative and computationally efficient. However, the asynchronous data format necessitates the development of specialized algorithms for event data processing. This research focused on event-based algorithm development for two system-level applications. The first application is event-based Shack-Hartmann wavefront sensing, a technique for measuring wavefront aberrations, whose use in adaptive optics relies on fast position tracking of an array of spots. These sensors conventionally use frame-based cameras operating at a fixed sampling rate to report pixel intensities, even though only a fraction of the pixels have signal. Prior in-lab experiments have shown feasibility of event-based cameras for Shack-Hartmann wavefront sensing (SHWFS), asynchronously reporting the spot locations as log intensity changes at microsecond time scale. This work proposes a novel convolutional neural network (CNN) called Event-Based Wavefront Network (EBWFNet) that achieves highly accurate estimation of spot centroid position in real-time. A custom Shack-Hartmann wavefront sensing hardware with common aperture for the synchronized frame- and event-based cameras was developed so that spot centroid locations computed from the frame-based camera may be used to train/test the event-CNN-based centroid position estimation method in an unsupervised manner. Field testing with this hardware shows that the proposed EBWFNet achieves sub-pixel accuracy in real-world scenarios with substantial improvement over the state-of-the-art event-based SHWFS. Continuing in SHWFS, this research proposes a second technique that uses a recurrent neural network (RNN), called Shack-Hartmann Event-Based RNN (SHEBRNN), that takes as input a stream of event data---pixel position, polarity, and time between events---and predicts the spot centroid position. The network was trained and evaluated with data from the custom SHWFS hardware that captures FBC and EBC simultaneously, using the spot centroid positions computed from the FBC as pseudo-ground truth to train/test the event-RNN-based centroid position estimation method in an unsupervised manner. The network improves the slope estimation accuracy and wavefront reconstruction Strehl ratio over the CNN-based state-of-the-art, while gaining substantial computational efficiency. By normalizing over the microlens's focal length, the network also achieves stable performance over various optical configurations. Finally, this research performs a feasibility study of event-based coded aperture with a microlens array to offload computationally expensive spatial processing to the optics prior to the EBC, and then use per-pixel RNNs to asynchronously update a learned hidden state for synchronous optical flow estimation. The technique is first demonstrated in simulation, and then a prototype experimental design is constructed to validate the approach. This prototype design is a proof-of-concept for future designs.

Keywords

Engineering

Comments

OCLC No. 1591813754

Rights Statement

Copyright 2026, author.

Link to Full Text

Share

COinS
 
 
 

Links