Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Last revision Both sides next revision
bottleneck_layer [2016/12/02 16:15]
127.0.0.1 external edit
bottleneck_layer [2018/04/23 18:40]
admin
Line 56: Line 56:
  
 We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method "Deep Variational Information Bottleneck",​ or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization,​ in terms of generalization performance and robustness to adversarial attack. We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method "Deep Variational Information Bottleneck",​ or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization,​ in terms of generalization performance and robustness to adversarial attack.
 +
 +https://​arxiv.org/​abs/​1804.07090 Low Rank Structure of Learned Representations
 +
 +In this paper, we study the dimensionality of the learned representations by models that have proved highly succesful for image classification. We focus on ResNet-18, ResNet-50 and VGG-19 and observe that when trained on CIFAR10 or CIFAR100 datasets, the learned representations exhibit a fairly low rank structure. We propose a modification to the training procedure, which further encourages low rank representations of activations at various stages in the neural network. Empirically,​ we show that this has implications for compression and robustness to adversarial examples.