Improvement and Implementation of Gumbel-Softmax VAE
Date of Award
2022
Degree Name
M.C.S. (Master of Computer Science)
Department
Department of Computer Science
Advisor/Chair
Zhongmei Yao
Abstract
Variational autoencoders (VAE) have recently become one of the most interesting developments in deep learning, as they take input data (e.g., images or text), learn its latent space, and then generate new similar and smooth data. The ability of discovering the latent space and creating new data makes VAEs powerful generative models, having applications in dimensionality reduction, data reconstruction, text automatic generation, art design, unsupervised clustering, semi-supervised classification, anomaly/outlier detection, and so on. Classic VAEs consider a Gaussian latent space, which does not allow for more complex representations. Gumbel-Softmax VAEs are an interesting extension, which provides practical solutions to implement the reparameterization trick for sampling a one-hot vector from a categorical distribution. During training, Gumbel-Softmax VAE needs to rely on softmax temperature tau, which guides the annealing process for categorical latent variables. Prior work simply decreases the temperature by a fixed factor and ignores the impact of the starting value and the active range of the temperature. We find that the temperature directly determines the performance of training. We present a novel parallel structure for VAEs, which combines two symmetric VAEs with different updating mechanics for the temperature and adjusts it at each training epoch based on the loss from these two VAEs. We show that our model offers a better performance than the original Gumbel-Softmax VAEs and can be used for data reconstruction, anomaly detection, and renovation of the imperfect with relatively lower distortion and noises.
Keywords
Computer Science, Gumbel-Softmax VAE, Softmax temperature, schemed of annealing temperature, parallel structure, anomaly detection
Rights Statement
Copyright © 2022, author.
Recommended Citation
Fangshi, Zhou, "Improvement and Implementation of Gumbel-Softmax VAE" (2022). Graduate Theses and Dissertations. 7168.
https://ecommons.udayton.edu/graduate_theses/7168
