Qi (Roger) She

GANs (3): Challenges in GANs

sheqi0399 — Mon, 11 Jun 2018 04:49:25 +0000

My prepared ppt for GANs review.

Classification of Problems and Related Paper:

First of all, read GANs tutorial from NIPS2016

1. Mode Collapse

The assumption of learning the full dataset well is not achieved, and only part of the dataset has been learned. The VAE can learned representation of the whole dataset, but not be very particular for each sample, while GANs can have sharp image, but encounter mode collapses more easily.

Unrolled GANs; (2017ICLR) **

Improves Techniques for Training GANs; (2016) ***

On Distinguishability Criteria for Estimating Generative Models; (2014) **
Towards Principled Methods for training GANs; (2016) ***

Mode Regularized GAN; (2016) **

Improved Training of Generative Adversarial Networks using Representative Features; (2018ICML) **

2. Non-convergence

We think it hard to achieve Nash equilibrium. Let me give a simple example for min-max two player game solved by simultaneous gradient descent methods.

On Distinguishability Criteria for Estimating Generative Models; (2014) **

Improves Techniques for Training GANs; (2016) ***

f-GAN: Training Generative Sampler using Variational Divergence Minimization; (2016)*** 

Stabilizing Training of Generative Adversarial Networks through Regularization; (2017) ** 

The Numerics of GANS; (2017NIPS) ** 

Gradient descent GAN optimization is locally stable; (2017NIPS) ** 

Which Training Methods for GANs do actually Converge?; (2018ICML,very new) ***

Improved Training of Generative Adversarial Networks using Representative Features; (2018ICML) **

3. Vanishing Gradient

When the final discriminator has been well trained, which means , when and , when . It will lead to the loss function goes to zero and the gradients are vanishing. Thus, for this challenge, it has two dilemmas:

Discriminator bad behavior, the Generator and loss function cannot represent the data distribution .
Discriminator good behavior, the gradient of loss function goes to zero, the learning is slow or fails.

Towards Principled Methods for training GANs; (2016) ***

Wasserstein GAN; (2017) ****

Improved Training of Wasserstein GANs; (2017NIPS) ***

4. Generalizability and Existence of Equilibrium

Generalization and Equilibrium in Generative Adversarial Nets (GANs); (2017) **

5. Evaluation methods for GANs

Improves Techniques for Training GANs; (2016) ***

Pros and Cons of GAN Evaluation Measures; (2018) *

Geometry Score: A Method For Comparing Generative Adversarial Networks

GANs (2): Summary of GAN theory paper (from my evernote)

sheqi0399 — Sun, 10 Jun 2018 04:10:44 +0000

Data	GANs paper	Problems they have	Solutions	Importance
2016/06	Improved Techniques for GANs	Overtraining of discriminator Generator Mode collaps GD not converge Vulnerable to adversarial sample	Feature matching Mini-batch discrimination Historical Averaging Label-smoothing	****
2016/06	f-GAN	No convergence to saddle point due to single step update	f-divergence; Variational approach of f-divergence	**
2016/11	Unrolled GAN	Mode collaps Oscillations between G and D	G’s consider future update of D	***
2017/01	Towards Principled Methods Training GANs	Optimal D leads to zero gradients when Distribution in low-dimensional manifold	Adding Gaussian noise	**
2017/03	W-GAN	Learning distribution is hard when they are actually in lower space	Define new distance considering to optimize	****
2017/05	Improved W-GAN	Lipschitz enforced by weight clipping.	Introduced a Gradient Penalty term.	***
2017/05	On the convergence and stability of GANs	GD is unstable leading to mode collapse; W-GAN and LS-GAN in domain space for regularization	Penalty term for sharp gradients	**
2017/11	Stabilizing Training of GANs through Regularization	density and dimensional mis-specifications high-dimension noise gives more variances in parameter estimation	convolving with noise noise induced regularization	**
2017/11	Gradient Descent GAN optimization is locally unstable	GAN optimization are not a convex-concave game thus having slow convergence and mode collapse problem	Even GAN optimization are not a convex-concave game but are locally asymptotically stable under proper condition Additional regularization term for gradient descent	***
2017/12	The Numerics of GANs	Gradients for two update problem fail due to eigenvalues have zero real part or large value of imaginary part	Consensus Optimization: Alternative method for finding new NASH Equailibrium Introduce norm of gradients	**
2018/05	Which Training Methods for GANs do actually Converge?	Unregularized GAN training is not always convergent.	GAN training with instance noise or zero centered gradient penalties converges. Propose simplified gradient penalties and prove local convergence dynamics	****
2018/05	Improved Training of Generative Adversarial Networks using Representative Features	Cannot have both image diversity and visual quality for original GANs	Pre-trained Autoencoder for feature extration from whole dataset, keeping whole distribution learned; combined with GANs	**
2018/05	On the Limitations of First-Order Approximation in GAN Dynamics	Learning dynamics are not yet well understood	Study GAN dynamics of vanishing gradients, mode collapse, and diverging or oscillatory behaviour An optimal discriminator provably converges, while first order approximations of D steps lead to unstable GAN dynamics and mode collapse	***

GANs (1): Some tracking on recent GANs

sheqi0399 — Sun, 03 Jun 2018 05:23:12 +0000

Recently this kind of DEEP METHOD is quite popular, below are some paper reading summary about GANs, which include some extensions for solving (1) vanishing gradients (W-GAN), (2) mode collapse (W-GAN, unrolled GAN), (3) diverging or oscillatory behavior (f-GAN), (4) existence of equilibrium and generalization and (5) evaluation of GAN performance. They result from (1) improper loss function and (2) optimization of the non-convex error function.

Although I am a fan of VAEs, I do think both Techniques are worth digging into. I will keep updated of this post. For GANs study, I may keep updated three aspects: (1) theory; (2) algorithm; and (3) application.

Theory

Original GANs:

Learn how to generate new samples from trained samples. Game theory leads to Nash Equilibrium, but in GANs do not know whether achieve it. The final results mean the generator produces the original samples. . Note discriminator as , Generator as . Simply two networks are competing with Minimax as follows:

For my understanding, by maximizing data log-likelihood which helps to formulate a good discriminator that distinguish from data and generator, while the second step minimizing the second term (only involved) equals to generate a good example from prior .

Some conclusions:

For fixed, the optimal is ;
The global minimum of is achieved if and only if and it is .

Algorithms:

During each training iterations

sample true data from
sample noise from prior
Get generated data from generator network
update discriminator using gradient descent as
Above learning discriminator network needs to be performed times

sample noise from prior
update generator now
Above learning generator network needs only once.

f-GAN

1. f-divergence uses the idea from Fenchel conjugate. It is a unified framework for GANs. That means it basically extends discriminator to any divergence belong to .

Two requirements are for : (1) should be convex function; (2) . Many examples of commonly-used divergence:

, KL-divergence
, Reverse-KL-divergence
, Chi-square

2. Fenchel conjugate

Every convex function has a conjugate function.

, which is basical idea of fenchel conjugate

3. Connection with original GANs

Where is a function whose input is and output is .

Thus finally we get

It is something similar and related to GANs

We need to figure out the transiformation function from to , which is the function of , thus

Borrowing from the idea of french conjugate, minmax is a saddle point optimization problem. Change the optimization in vanilla algorithm with single step update.

Wasserstein GAN:

This part has information learned from https://zhuanlan.zhihu.com/p/25071913, which is insightful and has more details, also from GAN to WGAN, good blog exists

For me, to some extend KO “DCGAN” paper discussed below

Solve unstable training problem
Solve collapse mode problem, making the samples generated to be more general
Give indicator like cross-entropy and accuracy to show the training process
No need careful deigning network structure, simple fully connected layer is ok

wasserstein reveals the distance that can truly tell us the distance between two distributions even they are not overlapping, while JS and KL cannot do it.

Basically, in WGAN,

(1) The Discriminator does not try to distinguish generated sample and the real sample, and they try to fit the Wasserstein distance among all the real samples, which is transferring from classification task to regression task;

(2) The Generator does try to generator sample like the real one instead they minimize the Wasserstein distance.

Improved Training of Wasserstein GANs:

WGAN still has non-convergence and low-quality generation problem, they think it results from the weight clipping method used in WGAN with Lipschitz constraint, thus they have penalized the norm of gradients of the critic with respect to its input. Which achieves stable training (no hyperparameter tuning) and high-quality generations.

My understanding of it: the weights clipping lead to weights to have very limited choices under training while gradient penalty (GP) improves it to enforce more diverse choices. When the weight changes sharply, it may lead to gradient explosion or vanishing, thus the key point is to find another way to meet Lipschitz constraint, which they call  gradient penalty (GP).

Info-GAN:

It is unsupervised learning method for disentangled representation. Basically, it is a framework by adding GANs with Mutual information method.

Intuition: Interpretable representation. By disentangling the features, it can be more clearly use them in the future task. In other words, we need to force a disentangled representation.

My understanding

Adding mutual information penalty into generator network, it means we intend to maximize the mutual information between latent code and to tell us that the latent codes modulate the generator results.
In the paper, that’s why the latent code has so much influence on the final generated images.
Forcing high information content, we encode the most interesting aspects of the representation into latent code .
latent code : meaningful information; noise input unrelated or unimportant information for images.

Instead of maximizing , can maximize the variational lower bound with approximated posterior

Unrolled GAN

Dynamic GAN:

The first paper introduces some theory on the convergence dynamics on GANs. The structure of discriminator network is analyzed through a Gaussian mixture model, which is a simple but intuitive way to have first-order dynamics analysis. The deep neural network dynamics are much more complex, and this step is really giving me insight into how the convergence dynamics influence to the final static state (people always focus on this aspect).

It also explains the problem of gradients vanishing and mode collapse problems in GANs. Also compared multiple step update on discriminator and the first-order update gives a theory of convergence dynamics on GMM structure.

The Numerics of GANs:

The paper proposes Numerics analysis of GANs, which suggests the convergence of GANs results from two principal features:

if the Jacobian has only eigenvalues with negative real-part at the equilibrium point, GAN training converges locally for small enough learning rates.
On the other hand, if the Jacobian has eigenvalues on the imaginary axis, it is generally not locally convergent

It is a more fundamental suggestion for designing a new algorithm. Absolutely continuous data and generator distributions have all eigenvalues of the
Jacobian have negative real-part, but it is not the commonly-seen case in the real world (where both distributions lie on lower dimensional manifolds).

Improves Techniques for Training GANs

Which Training Methods for GANs do actually Con verge?; (2018ICML) ***

I strongly recommend for this paper if you want to know more recent updates (until May 2018) on the convergence analysis of GAN gradient descent dynamics analysis.

Application

DCGAN:

Classification of Problems and Related Paper:

First of all, read GANs tutorial from NIPS2016

1. Mode Collapse

Unrolled GANs; (2017ICLR) **

Improves Techniques for Training GANs; (2016) ***

On Distinguishability Criteria for Estimating Generative Models; (2014) **
Towards Principled Methods for training GANs; (2016) ***

Mode Regularized GAN; (2016) **

Improved Training of Generative Adversarial Networks using Representative Features; (2018ICML) **

2. Non-convergence

On Distinguishability Criteria for Estimating Generative Models; (2014) **

Improves Techniques for Training GANs; (2016) ***

f-GAN: Training Generative Sampler using Variational Divergence Minimization; (2016)*** 

Stabilizing Training of Generative Adversarial Networks through Regularization; (2017) ** 

The Numerics of GANS; (2017NIPS) ** 

Gradient descent GAN optimization is locally stable; (2017NIPS) ** 

Which Training Methods for GANs do actually Converge?; (2018ICML,very new) ***

Improved Training of Generative Adversarial Networks using Representative Features; (2018ICML) **

3. Vanishing Gradient

Towards Principled Methods for training GANs; (2016) ***

Wasserstein GAN; (2017) ****

Improved Training of Wasserstein GANs; (2017NIPS) ***

4. Generalizability and Existence of Equilibrium

Generalization and Equilibrium in Generative Adversarial Nets (GANs); (2017) **

5. Evaluation methods for GANs

Improves Techniques for Training GANs; (2016) ***

Pros and Cons of GAN Evaluation Measures; (2018) *

Geometry Score: A Method For Comparing Generative Adversarial Networks

Interesting Blogs/Websites

sheqi0399 — Mon, 28 May 2018 09:31:30 +0000

http://www.inference.vc/: Posts on Machine Learning, Statistics from Ferenc Huszár.
https://csml.princeton.edu/readinggroup: CSML Reading Group from Princeton
https://casmls.github.io/: Columbia Advanced Machine Learning Seminar
https://towardsdatascience.com/: Towards data science, useful and clear blogs for data science
https://www.jiqizhixin.com/: China Jiqizhixin “机器之心” 专栏
https://zhuanlan.zhihu.com/jiqizhixin：”机器之心” 知乎专栏

Some notes about Tensorflow code (1)

sheqi0399 — Sat, 07 Apr 2018 06:49:27 +0000

These are some summary of my learning in Tensorflow. keep edited. Generally, for deep framework, we have three parts (yeah, that’s all) with

1. Loss function
2. Inference part: from to
3. Optimization

The codes look like

class AwesomeModel(object):
  def __init__(self):
    """ init the model with hyper-parameters etc """

  def inference(self, x):
    """ This is the forward calculation from x to y """
    return some_op(x, name="inference")

  def loss(self, batch_x, batch_y=None):
    y_predict = self.inference(batch_x)
    self.loss = tf.loss_function(y, y_predict, name="loss") # supervised
    # loss = tf.loss_function(x, y_predicted) # unsupervised

  def optimize(self, batch_x, batch_y):
    return tf.train.optimizer.minimize(self.loss, name="optimizer")

1. Saver

Our real-data takes so long, thus we need store it, and retrieve in the future. The variables constructed before Saver commend will be saved.

To store a computation graph, we can use

saver = tf.train.Saver()
saver.save(sess, checkpoints_file_name)

To restore a computation graph, we can use

saver = tf.train.import_meta_graph(checkpoints_file_name + '.meta')
saver.restore(sess, checkpoints_file_name)

2. tf.app.run() & FLAGS

Save the global variables in FLAGS, try to document your variables, and FLAGS have default documentation, it would save time when a lot of global variables.

tf.app.flags.DEFINE_boolean("some_flag", False, "Documentation")

FLAGS = tf.app.flags.FLAGS

def main(_):
  # your code goes here...
  # use FLAGS.some_flag in the code.

if __name__ == '__main__':
    tf.app.run()

3. Try to save operations name

we’d use

loss_tensor = tf.nn.softmax_cross_entropy_with_logits(logits, labels, dim=-1, name="loss")

to label the loss operation. We will no longer have loss_tensor when we restore the model, but we can always call graph.get_operation_by_name("loss") to get the operation.

4. Summaries

We should keep in mind that when coding , try to use summaries. Tensorflow provides the scaler, video, image summaries. We use summary like

# 1. Declare summaries that you'd like to collect.
tf.scalar_summary("summary_name", tensor, name = "summary_op_name")

# 2. Construct a summary writer object for the computation graph, once all summaries are defined.
summary_writer = tf.train.SummaryWriter(summary_dir_name, sess.graph)

# 3. Group all previously declared summaries for serialization. Usually we want all summaries defined
# in the computation graph. To pick a subset, use tf.merge_summary([summaries]).
summaries_tensor = tf.merge_all_summaries()

# 4. At runtime, in appropriate places, evaluate the summaries_tensor, to assign value.
summary_value, ... = sess.run([summaries_tensor, ...], feed_dict={...})

# 5. Write the summary value to disk, using summary writer.
summary_writer.add_summary(summary_value, global_step=step)

AI conference deadline tracking

sheqi0399 — Thu, 29 Mar 2018 13:29:08 +0000

AI conference deadline tracking

My brain words

sheqi0399 — Tue, 07 Nov 2017 23:52:41 +0000

These words have appeared in my research paper frequently ！～～～～ I suddenly want to know what happened in my research, thus it is natural to dig into the words frequently appeared in my publications. It shows that

Bayesian
Dynamic
Learning
Inference
Network
Model

……….

Hhh, I think it worth to keep tracking of it to see what I am really pursuing and fighting for!

Stochastic Optimization: the knowledge gradient

sheqi0399 — Wed, 25 Oct 2017 21:43:36 +0000

Today in Princeton 544 course, we have learned the knowledge gradient, and it is a one-period lookahead that maximizes the value of information: , where

is current belief state, is updated parameter estimates after running experiments;
is for choosing the best design given what we know;
is experiment proposed;
We do expectation for averaging over possible outcomes of the experiment (and our different belief about parameters)

This seems to be new for me, and less applicable for my current project, but I would like to introduce general contents below, which is summarized for four policies in all Stochastic optimization:

Policy function approximation
Parametric cost function approximation
Value function approximation
Direct lookheads

(1) and (2) need tunning for parameters, while it is more simple and easy to implement. and They are just parameter estimation in , for determining ; (3) and (4) need complex modeling.