### Sparse Autoencoder

Reference: Andrew Ng: Autoencoders and Sparsity

Autoencoder 的示意圖如下。主要目的是 output = decoder(encoder(input)) or

Our overall cost function is now:

where J(W,b) 是標準 L2 cost function.  beta 是 sparsity penalty weight.  rho 是 sparsity parameter, roh_hat 是 weight average value defined as follows:

be the average activation of hidden unit j (averaged over the training set). We would like to (approximately) enforce the constraint

Typically a small value close to zero (say 0.05). In other words, we would like the average activation of each hidden neuron j to be close to 0.05 (say).

### Stanford  Deep Learning Exercise: Sparse Autoencoder

Input image patch size: 8×8: 64 dimension.

Neural network 特性: 1 hidden layer; 25 hidden units; 0.01 sparsity parameter.

Input dimension: 64; hidden layer dimension: 25.   所以是降維。同時限制 sparsity parameter = 0.01.  (有用 L1 norm cost function?)

Sparsity training parameters:

lambda = 0.0001;   % weight decay parameter

beta = 3;       % weight of sparsity penalty

Just a refreshment, neural network 參數衆多，很容易 overfit.  常用的方法 to prevent overfitting:  regularization; drop out during training; pruning.  其中 pruning 其實就是 sparsity!

http://deeplearning.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity

https://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf

hiddenSize = 25; 下圖是 W1 25 weights (filters) 的圖示。

Appendix

 % sparseAutoencoderCost.m % calculate cost function and derivatives function [cost,grad] = sparseAutoencoderCost(theta, visibleSize, hiddenSize, ... lambda, sparsityParam, beta, data)   % visibleSize: the number of input units (probably 64) % hiddenSize: the number of hidden units (probably 25) % lambda: weight decay parameter % sparsityParam: The desired average activation for the hidden units (denoted in the lecture % notes by the greek alphabet rho, which looks like a lower-case "p"). % beta: weight of sparsity penalty term % data: Our 64x10000 matrix containing the training data. So, data(:,i) is the i-th training example.   % The input theta is a vector (because minFunc expects the parameters to be a vector). % We first convert theta to the (W1, W2, b1, b2) matrix/vector format, so that this % follows the notation convention of the lecture notes.   W1 = reshape(theta(1:hiddenSize*visibleSize), hiddenSize, visibleSize); W2 = reshape(theta(hiddenSize*visibleSize+1:2*hiddenSize*visibleSize), visibleSize, hiddenSize); b1 = theta(2*hiddenSize*visibleSize+1:2*hiddenSize*visibleSize+hiddenSize); b2 = theta(2*hiddenSize*visibleSize+hiddenSize+1:end);   % Cost and gradient variables (your code needs to compute these values). % Here, we initialize them to zeros. cost = 0; W1grad = zeros(size(W1)); W2grad = zeros(size(W2)); b1grad = zeros(size(b1)); b2grad = zeros(size(b2));   %% ---------- YOUR CODE HERE -------------------------------------- % Instructions: Compute the cost/optimization objective J_sparse(W,b) for the Sparse Autoencoder, % and the corresponding gradients W1grad, W2grad, b1grad, b2grad. % % W1grad, W2grad, b1grad and b2grad should be computed using backpropagation. % Note that W1grad has the same dimensions as W1, b1grad has the same dimensions % as b1, etc. Your code should set W1grad to be the partial derivative of J_sparse(W,b) with % respect to W1. I.e., W1grad(i,j) should be the partial derivative of J_sparse(W,b) % with respect to the input parameter W1(i,j). Thus, W1grad should be equal to the term % [(1/m) \Delta W^{(1)} + \lambda W^{(1)}] in the last block of pseudo-code in Section 2.2 % of the lecture notes (and similarly for W2grad, b1grad, b2grad). % % Stated differently, if we were using batch gradient descent to optimize the parameters, % the gradient descent update to W1 would be W1 := W1 - alpha * W1grad, and similarly for W2, b1, b2. %   [nFeatures, nSamples] = size(data); % first calculate the regular cost function J   [a1, a2, a3] = getActivation(W1, W2, b1, b2, data); errtp = ((a3 - data) .^ 2) ./ 2; err = sum(sum(errtp)) ./ nSamples; % now calculate pj which is the average activation of hidden units pj = sum(a2, 2) ./ nSamples; % the second part is weight decay part err2 = sum(sum(W1 .^ 2)) + sum(sum(W2 .^ 2)); err2 = err2 * lambda / 2; % the third part of overall cost function is the sparsity part err3 = zeros(hiddenSize, 1); err3 = err3 + sparsityParam .* log(sparsityParam ./ pj) + (1 - sparsityParam) .* log((1 - sparsityParam) ./ (1 - pj)); cost = err + err2 + beta * sum(err3);   % following are for calculating the grad of weights. delta3 = -(data - a3) .* dsigmoid(a3); delta2 = bsxfun(@plus, (W2' * delta3), beta .* (-sparsityParam ./ pj + (1 - sparsityParam) ./ (1 - pj))); delta2 = delta2 .* dsigmoid(a2); nablaW1 = delta2 * a1'; nablab1 = delta2; nablaW2 = delta3 * a2'; nablab2 = delta3;   W1grad = nablaW1 ./ nSamples + lambda .* W1; W2grad = nablaW2 ./ nSamples + lambda .* W2; b1grad = sum(nablab1, 2) ./ nSamples; b2grad = sum(nablab2, 2) ./ nSamples;   %------------------------------------------------------------------- % After computing the cost and gradient, we will convert the gradients back % to a vector format (suitable for minFunc). Specifically, we will unroll % your gradient matrices into a vector.   grad = [W1grad(:) ; W2grad(:) ; b1grad(:) ; b2grad(:)];   end   %------------------------------------------------------------------- % Here's an implementation of the sigmoid function, which you may find useful % in your computation of the costs and the gradients. This inputs a (row or % column) vector (say (z1, z2, z3)) and returns (f(z1), f(z2), f(z3)).   function sigm = sigmoid(x)   sigm = 1 ./ (1 + exp(-x)); end   %------------------------------------------------------------------- % This function calculate dSigmoid % function dsigm = dsigmoid(a) dsigm = a .* (1.0 - a);   end   %------------------------------------------------------------------- % This function return the activation of each layer % function [ainput, ahidden, aoutput] = getActivation(W1, W2, b1, b2, input)   ainput = input; ahidden = bsxfun(@plus, W1 * ainput, b1); ahidden = sigmoid(ahidden); aoutput = bsxfun(@plus, W2 * ahidden, b2); aoutput = sigmoid(aoutput); end
 % computeNumericalGradient.m % for the use of gradient check function numgrad = computeNumericalGradient(J, theta) % numgrad = computeNumericalGradient(J, theta) % theta: a vector of parameters % J: a function that outputs a real-number. Calling y = J(theta) will return the % function value at theta.   % Initialize numgrad with zeros numgrad = zeros(size(theta));   %% ---------- YOUR CODE HERE -------------------------------------- % Instructions: % Implement numerical gradient checking, and return the result in numgrad. % (See Section 2.3 of the lecture notes.) % You should write code so that numgrad(i) is (the numerical approximation to) the % partial derivative of J with respect to the i-th input argument, evaluated at theta. % I.e., numgrad(i) should be the (approximately) the partial derivative of J with % respect to theta(i). % % Hint: You will probably want to compute the elements of numgrad one at a time. size(theta) EPSILON = 1e-4; for i=1:size(theta) i memo = theta(i); theta(i) = memo + EPSILON; value1 = J(theta); theta(i) = memo - EPSILON; value2 = J(theta); theta(i) = memo; numgrad(i) = (value1 - value2) ./ (2 * EPSILON); end   %% --------------------------------------------------------------- end   
 % computeNumericalGradient.m % for the use of gradient check function numgrad = computeNumericalGradient(J, theta) % numgrad = computeNumericalGradient(J, theta) % theta: a vector of parameters % J: a function that outputs a real-number. Calling y = J(theta) will return the % function value at theta.   % Initialize numgrad with zeros numgrad = zeros(size(theta));   %% ---------- YOUR CODE HERE -------------------------------------- % Instructions: % Implement numerical gradient checking, and return the result in numgrad. % (See Section 2.3 of the lecture notes.) % You should write code so that numgrad(i) is (the numerical approximation to) the % partial derivative of J with respect to the i-th input argument, evaluated at theta. % I.e., numgrad(i) should be the (approximately) the partial derivative of J with % respect to theta(i). % % Hint: You will probably want to compute the elements of numgrad one at a time. size(theta) EPSILON = 1e-4; for i=1:size(theta) i memo = theta(i); theta(i) = memo + EPSILON; value1 = J(theta); theta(i) = memo - EPSILON; value2 = J(theta); theta(i) = memo; numgrad(i) = (value1 - value2) ./ (2 * EPSILON); end   %% --------------------------------------------------------------- end

 % sampleIMAGES.m % sampling patches for learning function patches = sampleIMAGES() % sampleIMAGES % Returns 10000 patches for training load IMAGES; % load images from disk patchsize = 8; % we'll use 8x8 patches numpatches = 10000; % Initialize patches with zeros. Your code will fill in this matrix--one % column per patch, 10000 columns. patches = zeros(patchsize*patchsize, numpatches);   %% ---------- YOUR CODE HERE -------------------------------------- % Instructions: Fill in the variable called "patches" using data % from IMAGES. % % IMAGES is a 3D array containing 10 images % For instance, IMAGES(:,:,6) is a 512x512 array containing the 6th image, % and you can type "imagesc(IMAGES(:,:,6)), colormap gray;" to visualize % it. (The contrast on these images look a bit off because they have % been preprocessed using using "whitening." See the lecture notes for % more details.) As a second example, IMAGES(21:30,21:30,1) is an image % patch corresponding to the pixels in the block (21,21) to (30,30) of % Image 1   counter = 1; ranimg = ceil(rand(1, numpatches) * 10); ranpix = ceil(rand(2, numpatches) * (512 - patchsize)); ranpixm = ranpix + patchsize - 1; while(counter <= numpatches) whichimg = ranimg(1, counter); whichpix = ranpix(:, counter); whichpixm = ranpixm(:, counter); patch = IMAGES(whichpix(1):whichpixm(1), whichpix(2):whichpixm(2), whichimg); repatch = reshape(patch, patchsize * patchsize, 1); patches(:, counter) = repatch; counter = counter + 1; end   %% --------------------------------------------------------------- % For the autoencoder to work well we need to normalize the data % Specifically, since the output of the network is bounded between [0,1] % (due to the sigmoid activation function), we have to make sure % the range of pixel values is also bounded between [0,1] patches = normalizeData(patches);   end   %% --------------------------------------------------------------- function patches = normalizeData(patches)   % Squash data to [0.1, 0.9] since we use sigmoid as the activation % function in the output layer   % Remove DC (mean of images). patches = bsxfun(@minus, patches, mean(patches)); % Truncate to +/-3 standard deviations and scale to -1 to 1 pstd = 3 * std(patches(:)); patches = max(min(patches, pstd), -pstd) / pstd; % Rescale from [-1,1] to [0.1,0.9] patches = (patches + 1) * 0.4 + 0.1;   end 
clc; close all;
%% CS294A/CS294W Programming Assignment Starter Code

%  Instructions
%  ------------
%
%  This file contains code that helps you get started on the
%  programming assignment. You will need to complete the code in sampleIMAGES.m,
%  For the purpose of completing the assignment, you do not need to
%  change the code in this file.
%
%%======================================================================
%% STEP 0: Here we provide the relevant parameters values that will
%  allow your sparse autoencoder to get good filters; you do not need to
%  change the parameters below.

visibleSize = 8*8;   % number of input units
hiddenSize = 25;     % number of hidden units
sparsityParam = 0.01;   % desired average activation of the hidden units.
% (This was denoted by the Greek alphabet rho, which looks like a lower-case "p",
%  in the lecture notes).
lambda = 0.0001;     % weight decay parameter
beta = 3;            % weight of sparsity penalty term

%%======================================================================
%% STEP 1: Implement sampleIMAGES
%
%  After implementing sampleIMAGES, the display_network command should
%  display a random sample of 200 patches from the dataset

patches = sampleIMAGES;

display_network(patches(:,randi(size(patches,2),200,1)),8);

% %just for debug
% lambda = 0;
% beta = 0;
% hiddenSize = 2;
% patches = patches(:, 1:10);

%  Obtain random parameters theta
theta = initializeParameters(hiddenSize, visibleSize);

%%======================================================================
%% STEP 2: Implement sparseAutoencoderCost
%
%  You can implement all of the components (squared error cost, weight decay term,
%  sparsity penalty) in the cost function at once, but it may be easier to do
%  it step-by-step and run gradient checking (see STEP 3) after each step.  We
%  suggest implementing the sparseAutoencoderCost function using the following steps:
%
%  (a) Implement forward propagation in your neural network, and implement the
%      squared error term of the cost function.  Implement backpropagation to
%      compute the derivatives.   Then (using lambda=beta=0), run Gradient Checking
%      to verify that the calculations corresponding to the squared error cost
%      term are correct.
%
%  (b) Add in the weight decay term (in both the cost function and the derivative
%      calculations), then re-run Gradient Checking to verify correctness.
%
%  (c) Add in the sparsity penalty term, then re-run Gradient Checking to
%      verify correctness.
%
%  Feel free to change the training settings when debugging your
%  code.  (For example, reducing the training set size or
%  number of hidden units may make your code run faster; and setting beta
%  and/or lambda to zero may be helpful for debugging.)  However, in your
%  final submission of the visualized weights, please use parameters we
%  gave in Step 0 above.
[cost, grad] = sparseAutoencoderCost(theta, visibleSize, hiddenSize, lambda, ...
sparsityParam, beta, patches);

%%======================================================================
%
% Hint: If you are debugging your code, performing gradient checking on smaller models
% and smaller training sets (e.g., using only 10 training examples and 1-2 hidden
% units) may speed things up.

% First, lets make sure your numerical gradient computation is correct for a
% simple function.  After you have implemented computeNumericalGradient.m,
% run the following:

% % Now we can use it to check your cost function and derivative calculations
% % for the sparse autoencoder.
%                                                   hiddenSize, lambda, ...
%                                                   sparsityParam, beta, ...
%                                                   patches), theta);
%
% % Use this to visually compare the gradients side by side
%
% % Compare numerically computed gradients with the ones obtained from backpropagation
% disp(diff); % Should be small. In our implementation, these values are
%             % usually less than 1e-9.
%
%             % When you got this working, Congratulations!!!

%%======================================================================
%% STEP 4: After verifying that your implementation of
%  sparseAutoencoderCost is correct, You can start training your sparse
%  autoencoder with minFunc (L-BFGS).

%  Randomly initialize the parameters
theta = initializeParameters(hiddenSize, visibleSize);

%  Use minFunc to minimize the function
options.Method = 'lbfgs'; % Here, we use L-BFGS to optimize our cost
% function. Generally, for minFunc to work, you
% need a function pointer with two outputs: the
% function value and the gradient. In our problem,
% sparseAutoencoderCost.m satisfies this.
options.maxIter = 400;	  % Maximum number of iterations of L-BFGS to run
options.display = 'on';

[opttheta, cost] = minFunc( @(p) sparseAutoencoderCost(p, ...
visibleSize, hiddenSize, ...
lambda, sparsityParam, ...
beta, patches), ...
theta, options);

%%======================================================================
%% STEP 5: Visualization

W1 = reshape(opttheta(1:hiddenSize*visibleSize), hiddenSize, visibleSize);
display_network(W1', 12);

print -djpeg weights.jpg   % save the visualization to a file