Autoencoder in Keras

by allenlu2007


前(數)文討論各類 autoencoder (fully connected autoencoder, sparse autoencoder, multi-layer fully connected autoencoder, convolutional autoencoder, denoise autoencoder, variational autoencoder).

剛好 reference 用 Keras 重新整理以上的 autoencoders.

  • a simple autoencoder based on a fully-connected layer
  • a sparse autoencoder
  • a deep fully-connected autoencoder
  • a deep convolutional autoencoder
  • an image denoising model
  • a sequence-to-sequence autoencoder
  • a variational autoencoder

What are autoencoders?

Autoencoder: schema

“Autoencoding” is a data compression algorithm where the compression and decompression functions are 1) data-specific, 2) lossy, and 3) learned automatically from examples rather than engineered by a human. Additionally, in almost all contexts where the term “autoencoder” is used, the compression and decompression functions are implemented with neural networks.

本文主要是從 compression 角度出發。先說結論:autoencoder 不適合做一般普適性 compression algorithm (like JPEG/MPEG etc.).  

1. Pretrain deep learning network

原先 autoencoder 似乎沒有太實際的用途。在 2012 (Hinton?) 發現 autoencoder 非常適合 deep convolutional neural network 的 greedy layer-wise pretraining!  

前文說到 deep neural network 如何避免陷入 local minimum.  Greedy layer-wise pretraining 似乎可以避免 local minimum. 

不過之後發現 random initial weights 似乎效果也不錯。因此後來也不用 autoencoder 做 pretraining.  2014 batch normalization; 2015 一般開始用 residual learning to train deep networks from scratch.


2. Dimension reduction for visualization

 t-SNE 算是目前最好的 2D visualization algorithm.  但是 t-SNE 一般需要低維的 input data.  一個好的策略是 autoencoder + t-SNE.

先用 autoencoder 把高維的 data 壓縮到低維 (e.g. 32 dimensional), 然後再用 t-SNE map 壓縮 data 到 2D plane.  

3. Denoise