Sparse Autoencoder

by allenlu2007

Reference: Andrew Ng: Autoencoders and Sparsity

Autoencoder 的示意圖如下。主要目的是 output = decoder(encoder(input)) or

\textstyle h_{W,b}(x) \approx x

 image

前文討論 autoencoder, unsupervised learning, 主要的目的是降維,也就是 L2 dimension 小於 L1 dimension.  另一個相反的做法不是降維,而是 explore data sparsity feature. 甚至可以 increase dimension (L2 > L1).  但是引入新的 constraint, sparsity.   Sparsity 一方面是一種 (external) data representation (e.g. like one hot representation); 另一方面是 explore data intrinsic structure. 

如何引入 sparsity constraint?  一個從 convex optimization 學到的方法就是用 L1 norm 作爲 cost function, 加上或是取代原來的 L2 norm cost function, e.g. Lasso penalty, TV (Total Variation) denoising/regularization or compressed sensing.

在 neural network or heavily reply on back propagation 的網路,似乎都 prefer 用 L2 而避免用 L1 cost function.  我認爲主要是 L2 differentiable and back propagation error term 很簡單。

因此 neural network 用另一個方法是 sparsity parameter ~ 0, 也就是 weight average 值。搭配 nonlinear function sigmoid (0-1), 得到 sparsity 的效果。此時 cost function 仍然使用 L2 norm.  衹是增加一個額外的 sparsity penalty cost!!

Our overall cost function is now:

\begin{align}
J_{\rm sparse}(W,b) = J(W,b) + \beta \sum_{j=1}^{s_2} {\rm KL}(\rho || \hat\rho_j),
\end{align}

where J(W,b) 是標準 L2 cost function.  beta 是 sparsity penalty weight.  rho 是 sparsity parameter, roh_hat 是 weight average value defined as follows:

\begin{align}
\hat\rho_j = \frac{1}{m} \sum_{i=1}^m \left[ a^{(2)}_j(x^{(i)}) \right]
\end{align}

be the average activation of hidden unit j (averaged over the training set). We would like to (approximately) enforce the constraint

\begin{align}
\hat\rho_j = \rho,
\end{align}

Typically a small value close to zero (say 0.05). In other words, we would like the average activation of each hidden neuron j to be close to 0.05 (say). 

如何限制 sparsity constraint, 就是找到一個距離函數 d(rho, rho_hat) 作爲 penalty term.  在統計上衡量兩個分佈的距離常用的函數就是 Kullback-Leibler divergence (KL) as follow:

 \begin{align}
\sum_{j=1}^{s_2} \rho \log \frac{\rho}{\hat\rho_j} + (1-\rho) \log \frac{1-\rho}{1-\hat\rho_j}.
\end{align}

爲什麽要用 KL divergence? 其他的 distance function 例如 square, abs, etc. 是否可用? KL divergence 帶有統計的意義 (Baysian?)  KL function (rho=0.2) 如下。最小值是在 rho_hat = rho = 0.2, 但在 rho_hat = 0 or 1 時爲無限大。

KLPenaltyExample.png

 

Stanford  Deep Learning Exercise: Sparse Autoencoder

最原始的數據是: 512×512 size 灰階的影像,共有十張。其中第六張如下。

image

再來 matlab autoencoder program 把影像分割爲 size 8×8 的 patches. 共有 (512/8)x(512/8) x 10 = 64x64x10 = 40960 patches. 

以上是我想的做法。顯然 matlab code 不是如此實現。而是隨機挑選其中 10000 patch.  而且每一個 patch 的坐標也是隨機選取。一些 sample patches (共 200 個)如下。 同時爲了 neural network 的非綫性 sigmoid 特性,再做 patch normalizeData to 0-1.

Input image patch size: 8×8: 64 dimension.

image

Neural network 特性: 1 hidden layer; 25 hidden units; 0.01 sparsity parameter.

Input dimension: 64; hidden layer dimension: 25.   所以是降維。同時限制 sparsity parameter = 0.01.  (有用 L1 norm cost function?)

Sparsity training parameters:

lambda = 0.0001;   % weight decay parameter

beta = 3;       % weight of sparsity penalty

因爲 cost function 包含 L2 cost 以及 sparsity cost. 

 \begin{align}
J_{\rm sparse}(W,b) = J(W,b) + \beta \sum_{j=1}^{s_2} {\rm KL}(\rho || \hat\rho_j),
\end{align}
所以 back propagation (error) function (i.e. directional derivative) 也包含兩項。KL sparsity cost 的 derivative 非常簡單,就是下式右手第二項。superscript 的 2 代表 neural network 的 layer2.
\begin{align}
\delta^{(2)}_i =
  \left( \left( \sum_{j=1}^{s_{2}} W^{(2)}_{ji} \delta^{(3)}_j \right)
+ \beta \left( - \frac{\rho}{\hat\rho_i} + \frac{1-\rho}{1-\hat\rho_i} \right) \right) f'(z^{(2)}_i) .
\end{align}
檢識 matlab code sparseAutoencoderCost.m 發現有三項 error term: err 對應第一項,err3 對應第二項。多了 err2 其中有 weight decay parameter lambda, why? 請參考 Andrew Ng CS294a 完整的 lecture note: sparseAutoencoder.pdf.  基本是 weights L2 regularization term to prevent overfitting!   這項是 neural network back propagation 就會用到的term, 和 sparsity 無關。

Just a refreshment, neural network 參數衆多,很容易 overfit.  常用的方法 to prevent overfitting:  regularization; drop out during training; pruning.  其中 pruning 其實就是 sparsity!

http://deeplearning.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity

https://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf

hiddenSize = 25; 下圖是 W1 25 weights (filters) 的圖示。

image

 

Appendix

 

% sparseAutoencoderCost.m
% calculate cost function and derivatives
function [cost,grad] = sparseAutoencoderCost(theta, visibleSize, hiddenSize, ...
lambda, sparsityParam, beta, data)
 
% visibleSize: the number of input units (probably 64) 
% hiddenSize: the number of hidden units (probably 25) 
% lambda: weight decay parameter
% sparsityParam: The desired average activation for the hidden units (denoted in the lecture
% notes by the greek alphabet rho, which looks like a lower-case "p").
% beta: weight of sparsity penalty term
% data: Our 64x10000 matrix containing the training data. So, data(:,i) is the i-th training example. 
 
% The input theta is a vector (because minFunc expects the parameters to be a vector). 
% We first convert theta to the (W1, W2, b1, b2) matrix/vector format, so that this 
% follows the notation convention of the lecture notes.
 
W1 = reshape(theta(1:hiddenSize*visibleSize), hiddenSize, visibleSize);
W2 = reshape(theta(hiddenSize*visibleSize+1:2*hiddenSize*visibleSize), visibleSize, hiddenSize);
b1 = theta(2*hiddenSize*visibleSize+1:2*hiddenSize*visibleSize+hiddenSize);
b2 = theta(2*hiddenSize*visibleSize+hiddenSize+1:end);
 
% Cost and gradient variables (your code needs to compute these values). 
% Here, we initialize them to zeros. 
cost = 0;
W1grad = zeros(size(W1)); 
W2grad = zeros(size(W2));
b1grad = zeros(size(b1)); 
b2grad = zeros(size(b2));
 
%% ---------- YOUR CODE HERE --------------------------------------
% Instructions: Compute the cost/optimization objective J_sparse(W,b) for the Sparse Autoencoder,
% and the corresponding gradients W1grad, W2grad, b1grad, b2grad.
%
% W1grad, W2grad, b1grad and b2grad should be computed using backpropagation.
% Note that W1grad has the same dimensions as W1, b1grad has the same dimensions
% as b1, etc. Your code should set W1grad to be the partial derivative of J_sparse(W,b) with
% respect to W1. I.e., W1grad(i,j) should be the partial derivative of J_sparse(W,b) 
% with respect to the input parameter W1(i,j). Thus, W1grad should be equal to the term 
% [(1/m) \Delta W^{(1)} + \lambda W^{(1)}] in the last block of pseudo-code in Section 2.2 
% of the lecture notes (and similarly for W2grad, b1grad, b2grad).
% 
% Stated differently, if we were using batch gradient descent to optimize the parameters,
% the gradient descent update to W1 would be W1 := W1 - alpha * W1grad, and similarly for W2, b1, b2. 
%
 
[nFeatures, nSamples] = size(data);
% first calculate the regular cost function J
 
[a1, a2, a3] = getActivation(W1, W2, b1, b2, data);
errtp = ((a3 - data) .^ 2) ./ 2;
err = sum(sum(errtp)) ./ nSamples;
% now calculate pj which is the average activation of hidden units
pj = sum(a2, 2) ./ nSamples;
% the second part is weight decay part
err2 = sum(sum(W1 .^ 2)) + sum(sum(W2 .^ 2));
err2 = err2 * lambda / 2;
% the third part of overall cost function is the sparsity part
err3 = zeros(hiddenSize, 1);
err3 = err3 + sparsityParam .* log(sparsityParam ./ pj) + (1 - sparsityParam) .* log((1 - sparsityParam) ./ (1 - pj));
cost = err + err2 + beta * sum(err3);
 
% following are for calculating the grad of weights.
delta3 = -(data - a3) .* dsigmoid(a3);
delta2 = bsxfun(@plus, (W2' * delta3), beta .* (-sparsityParam ./ pj + (1 - sparsityParam) ./ (1 - pj))); 
delta2 = delta2 .* dsigmoid(a2);
nablaW1 = delta2 * a1';
nablab1 = delta2;
nablaW2 = delta3 * a2';
nablab2 = delta3;
 
W1grad = nablaW1 ./ nSamples + lambda .* W1;
W2grad = nablaW2 ./ nSamples + lambda .* W2;
b1grad = sum(nablab1, 2) ./ nSamples;
b2grad = sum(nablab2, 2) ./ nSamples;
 
%-------------------------------------------------------------------
% After computing the cost and gradient, we will convert the gradients back
% to a vector format (suitable for minFunc). Specifically, we will unroll
% your gradient matrices into a vector.
 
grad = [W1grad(:) ; W2grad(:) ; b1grad(:) ; b2grad(:)];
 
end
 
%-------------------------------------------------------------------
% Here's an implementation of the sigmoid function, which you may find useful
% in your computation of the costs and the gradients. This inputs a (row or
% column) vector (say (z1, z2, z3)) and returns (f(z1), f(z2), f(z3)).
 
function sigm = sigmoid(x)
 
sigm = 1 ./ (1 + exp(-x));
end
 
%-------------------------------------------------------------------
% This function calculate dSigmoid
%
function dsigm = dsigmoid(a)
dsigm = a .* (1.0 - a);
 
end
 
%-------------------------------------------------------------------
% This function return the activation of each layer
%
function [ainput, ahidden, aoutput] = getActivation(W1, W2, b1, b2, input)
 
ainput = input;
ahidden = bsxfun(@plus, W1 * ainput, b1);
ahidden = sigmoid(ahidden);
aoutput = bsxfun(@plus, W2 * ahidden, b2);
aoutput = sigmoid(aoutput);
end
% computeNumericalGradient.m
% for the use of gradient check
function numgrad = computeNumericalGradient(J, theta)
% numgrad = computeNumericalGradient(J, theta)
% theta: a vector of parameters
% J: a function that outputs a real-number. Calling y = J(theta) will return the
% function value at theta. 
 
% Initialize numgrad with zeros
numgrad = zeros(size(theta));
 
%% ---------- YOUR CODE HERE --------------------------------------
% Instructions: 
% Implement numerical gradient checking, and return the result in numgrad. 
% (See Section 2.3 of the lecture notes.)
% You should write code so that numgrad(i) is (the numerical approximation to) the 
% partial derivative of J with respect to the i-th input argument, evaluated at theta. 
% I.e., numgrad(i) should be the (approximately) the partial derivative of J with 
% respect to theta(i).
% 
% Hint: You will probably want to compute the elements of numgrad one at a time. 
size(theta)
EPSILON = 1e-4;
for i=1:size(theta)
i
memo = theta(i);
theta(i) = memo + EPSILON;
value1 = J(theta);
theta(i) = memo - EPSILON;
value2 = J(theta);
theta(i) = memo;
numgrad(i) = (value1 - value2) ./ (2 * EPSILON);
end
 
%% ---------------------------------------------------------------
end
 
% computeNumericalGradient.m
% for the use of gradient check
function numgrad = computeNumericalGradient(J, theta)
% numgrad = computeNumericalGradient(J, theta)
% theta: a vector of parameters
% J: a function that outputs a real-number. Calling y = J(theta) will return the
% function value at theta. 
 
% Initialize numgrad with zeros
numgrad = zeros(size(theta));
 
%% ---------- YOUR CODE HERE --------------------------------------
% Instructions: 
% Implement numerical gradient checking, and return the result in numgrad. 
% (See Section 2.3 of the lecture notes.)
% You should write code so that numgrad(i) is (the numerical approximation to) the 
% partial derivative of J with respect to the i-th input argument, evaluated at theta. 
% I.e., numgrad(i) should be the (approximately) the partial derivative of J with 
% respect to theta(i).
% 
% Hint: You will probably want to compute the elements of numgrad one at a time. 
size(theta)
EPSILON = 1e-4;
for i=1:size(theta)
i
memo = theta(i);
theta(i) = memo + EPSILON;
value1 = J(theta);
theta(i) = memo - EPSILON;
value2 = J(theta);
theta(i) = memo;
numgrad(i) = (value1 - value2) ./ (2 * EPSILON);
end
 
%% ---------------------------------------------------------------
end

 

% sampleIMAGES.m
% sampling patches for learning
function patches = sampleIMAGES()
% sampleIMAGES
% Returns 10000 patches for training
load IMAGES; % load images from disk 
patchsize = 8; % we'll use 8x8 patches 
numpatches = 10000;
% Initialize patches with zeros. Your code will fill in this matrix--one
% column per patch, 10000 columns. 
patches = zeros(patchsize*patchsize, numpatches);
 
%% ---------- YOUR CODE HERE --------------------------------------
% Instructions: Fill in the variable called "patches" using data 
% from IMAGES. 
% 
% IMAGES is a 3D array containing 10 images
% For instance, IMAGES(:,:,6) is a 512x512 array containing the 6th image,
% and you can type "imagesc(IMAGES(:,:,6)), colormap gray;" to visualize
% it. (The contrast on these images look a bit off because they have
% been preprocessed using using "whitening." See the lecture notes for
% more details.) As a second example, IMAGES(21:30,21:30,1) is an image
% patch corresponding to the pixels in the block (21,21) to (30,30) of
% Image 1
 
counter = 1;
ranimg = ceil(rand(1, numpatches) * 10);
ranpix = ceil(rand(2, numpatches) * (512 - patchsize));
ranpixm = ranpix + patchsize - 1;
while(counter <= numpatches)
whichimg = ranimg(1, counter);
whichpix = ranpix(:, counter);
whichpixm = ranpixm(:, counter);
patch = IMAGES(whichpix(1):whichpixm(1), whichpix(2):whichpixm(2), whichimg);
repatch = reshape(patch, patchsize * patchsize, 1);
patches(:, counter) = repatch;
counter = counter + 1;
end
 
%% ---------------------------------------------------------------
% For the autoencoder to work well we need to normalize the data
% Specifically, since the output of the network is bounded between [0,1]
% (due to the sigmoid activation function), we have to make sure 
% the range of pixel values is also bounded between [0,1]
patches = normalizeData(patches);
 
end
 
%% ---------------------------------------------------------------
function patches = normalizeData(patches)
 
% Squash data to [0.1, 0.9] since we use sigmoid as the activation
% function in the output layer
 
% Remove DC (mean of images). 
patches = bsxfun(@minus, patches, mean(patches));
% Truncate to +/-3 standard deviations and scale to -1 to 1
pstd = 3 * std(patches(:));
patches = max(min(patches, pstd), -pstd) / pstd;
% Rescale from [-1,1] to [0.1,0.9]
patches = (patches + 1) * 0.4 + 0.1;
 
end

clc; close all;
%% CS294A/CS294W Programming Assignment Starter Code

%  Instructions
%  ------------
% 
%  This file contains code that helps you get started on the
%  programming assignment. You will need to complete the code in sampleIMAGES.m,
%  sparseAutoencoderCost.m and computeNumericalGradient.m. 
%  For the purpose of completing the assignment, you do not need to
%  change the code in this file. 
%
%%======================================================================
%% STEP 0: Here we provide the relevant parameters values that will
%  allow your sparse autoencoder to get good filters; you do not need to 
%  change the parameters below.

visibleSize = 8*8;   % number of input units 
hiddenSize = 25;     % number of hidden units 
sparsityParam = 0.01;   % desired average activation of the hidden units.
                     % (This was denoted by the Greek alphabet rho, which looks like a lower-case "p",
		     %  in the lecture notes). 
lambda = 0.0001;     % weight decay parameter       
beta = 3;            % weight of sparsity penalty term       

%%======================================================================
%% STEP 1: Implement sampleIMAGES
%
%  After implementing sampleIMAGES, the display_network command should
%  display a random sample of 200 patches from the dataset

patches = sampleIMAGES;

display_network(patches(:,randi(size(patches,2),200,1)),8);

% %just for debug
% lambda = 0;
% beta = 0;
% hiddenSize = 2;
% patches = patches(:, 1:10);


%  Obtain random parameters theta
theta = initializeParameters(hiddenSize, visibleSize);

%%======================================================================
%% STEP 2: Implement sparseAutoencoderCost
%
%  You can implement all of the components (squared error cost, weight decay term,
%  sparsity penalty) in the cost function at once, but it may be easier to do 
%  it step-by-step and run gradient checking (see STEP 3) after each step.  We 
%  suggest implementing the sparseAutoencoderCost function using the following steps:
%
%  (a) Implement forward propagation in your neural network, and implement the 
%      squared error term of the cost function.  Implement backpropagation to 
%      compute the derivatives.   Then (using lambda=beta=0), run Gradient Checking 
%      to verify that the calculations corresponding to the squared error cost 
%      term are correct.
%
%  (b) Add in the weight decay term (in both the cost function and the derivative
%      calculations), then re-run Gradient Checking to verify correctness. 
%
%  (c) Add in the sparsity penalty term, then re-run Gradient Checking to 
%      verify correctness.
%
%  Feel free to change the training settings when debugging your
%  code.  (For example, reducing the training set size or 
%  number of hidden units may make your code run faster; and setting beta 
%  and/or lambda to zero may be helpful for debugging.)  However, in your 
%  final submission of the visualized weights, please use parameters we 
%  gave in Step 0 above.
[cost, grad] = sparseAutoencoderCost(theta, visibleSize, hiddenSize, lambda, ...
                                     sparsityParam, beta, patches);

%%======================================================================
%% STEP 3: Gradient Checking
%
% Hint: If you are debugging your code, performing gradient checking on smaller models 
% and smaller training sets (e.g., using only 10 training examples and 1-2 hidden 
% units) may speed things up.

% First, lets make sure your numerical gradient computation is correct for a
% simple function.  After you have implemented computeNumericalGradient.m,
% run the following: 
checkNumericalGradient();

% % Now we can use it to check your cost function and derivative calculations
% % for the sparse autoencoder.  
% numgrad = computeNumericalGradient( @(x) sparseAutoencoderCost(x, visibleSize, ...
%                                                   hiddenSize, lambda, ...
%                                                   sparsityParam, beta, ...
%                                                   patches), theta);
% 
% % Use this to visually compare the gradients side by side
% disp([numgrad grad]); 
% 
% % Compare numerically computed gradients with the ones obtained from backpropagation
% diff = norm(numgrad-grad)/norm(numgrad+grad);
% disp(diff); % Should be small. In our implementation, these values are
%             % usually less than 1e-9.
% 
%             % When you got this working, Congratulations!!! 

%%======================================================================
%% STEP 4: After verifying that your implementation of
%  sparseAutoencoderCost is correct, You can start training your sparse
%  autoencoder with minFunc (L-BFGS).

%  Randomly initialize the parameters
theta = initializeParameters(hiddenSize, visibleSize);

%  Use minFunc to minimize the function
addpath minFunc/
options.Method = 'lbfgs'; % Here, we use L-BFGS to optimize our cost
                          % function. Generally, for minFunc to work, you
                          % need a function pointer with two outputs: the
                          % function value and the gradient. In our problem,
                          % sparseAutoencoderCost.m satisfies this.
options.maxIter = 400;	  % Maximum number of iterations of L-BFGS to run 
options.display = 'on';


[opttheta, cost] = minFunc( @(p) sparseAutoencoderCost(p, ...
                                   visibleSize, hiddenSize, ...
                                   lambda, sparsityParam, ...
                                   beta, patches), ...
                                   theta, options);

%%======================================================================
%% STEP 5: Visualization 

W1 = reshape(opttheta(1:hiddenSize*visibleSize), hiddenSize, visibleSize);
display_network(W1', 12); 

print -djpeg weights.jpg   % save the visualization to a file 

Advertisements