annealed

Implementation and reproducing paper "Improving Explorability in Variational Inference with Annealed Variational Objectives Bayesian methods" https://arxiv.org/abs/1809.01818

Team

• Aleksei Pronkin
• Mikhail Kurenkov
• Timur Chikichev

Proposal

see: Proposal in this repo.

Reproducing

Training VAE

For reproducing experiments with VAE use following commands

python3 run_vae.py -epoch 100 train vae_hvi

Epoch is a number of epoch for training. vae_hvi is a possible model. Other possible models are vae, vae_hvi, vae_hvi_avo. Logs is saved to lightning_logs. Checkpoint is saved to lightning_logs/versrion_{version_number}/checkpoints.

Evaluation VAE

For testing VAE use following command

python run_vae.py -model-checkpoint pretrained_models/vae_hvi_avo.ckpt test vae_hvi_avo

This command also generate generation.png and reconstruction.png

Goal

During this project we are going to implement methods and repeat experiments of this paper. https://arxiv.org/abs/1809.01818

Experiments

• Biased noise model.
• Toy energy fitting.
• Quantitative analysis on robustness to beta annealing.
• Amortized inference on MNIST dataset.

Results

Annealed toy distributions:

Toy distributions experiments:

Reconstruction of VAE. Left is ground truth, right is reconstructed.

Models

• Variational Auto Encoder (VAE) https://arxiv.org/abs/1312.6114 with parameters from https://github.com/jmtomczak/vae_householder_flow
• Hierarchical Variational Model (HVM) based on https://arxiv.org/abs/1511.02386.
• Importance Weighted Auto-Encoder (IWAE) based on https://arxiv.org/abs/1509.00519.

Pipeline

• Implement models.
• Implement AVO method https://arxiv.org/abs/1809.01818.
• Implement Hierarchical Variational Inference (HVI) method https://arxiv.org/abs/1511.02386.
• Analyze a behavior of AVO objective on toy examples.
• Conduct experiments with VAE and AVO on the MNIST and CelebA datasets.

We planned to use and rewrite some code from https://github.com/joelouismarino/iterative_inference/, https://github.com/jmtomczak/vae_householder_flow, https://github.com/AntixK/PyTorch-VAE, https://github.com/haofuml/cyclical_annealing and https://github.com/ajayjain/lmconv. (We assume, that first two repositories were used in the original paper closed source code)