This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
invariant_representation [2018/04/14 00:34]
invariant_representation [2018/12/21 18:19]
Line 357: Line 357:
 We introduce a Group Convolutional Neural Network with linear equivariance to translations and right angle rotations in three dimensions. We call this network CubeNet, reflecting its cube-like symmetry. By construction,​ this network helps preserve a 3D shape'​s global and local signature, as it is transformed through successive layers. We introduce a Group Convolutional Neural Network with linear equivariance to translations and right angle rotations in three dimensions. We call this network CubeNet, reflecting its cube-like symmetry. By construction,​ this network helps preserve a 3D shape'​s global and local signature, as it is transformed through successive layers.
 +Another perspective on our approach is to think of it as global average pooling
 +over rotations, where we expose a new ‘rotation-dimension.’ Without adhering to a
 +defined group, it would be challenging to disentangle or orient a feature space (at any
 +one layer, or across multiple layers) with respect to such a rotation dimension. ​
 +https://​arxiv.org/​abs/​1803.00502v3 PIP Distance: A Unitary-invariant Metric for Understanding Functionality and Dimensionality of Vector Embeddings
 +With tools from perturbation and stability theory, we provide an upper bound on the PIP loss using the signal spectrum and noise variance, both of which can be readily inferred from data. Our framework sheds light on many empirical phenomena, including the existence of an optimal dimension, and the robustness of embeddings against over-parametrization. The bias-variance tradeoff of PIP loss explicitly answers the fundamental open problem of dimensionality selection for vector embeddings. ​ https://​github.com/​aaaasssddf/​PIP-experiments
 +In this paper, we introduce a mathematically sound theory for vector
 +embeddings, from a stability point of view. Our theory answers some open
 +questions, in particular:
 +1. What is an appropriate metric for comparing different vector embeddings?
 +2. How to select dimensionality for vector embeddings?
 +3. Why people choose different dimensionalities but they all work well in
 +We present a theoretical analysis for embeddings starting from first principles.
 +We first propose a novel objective, the Pairwise Inner Product (PIP)
 +loss. The PIP loss is closely related to the functionality differences between
 +the embeddings, and a small PIP loss means the two embeddings are close
 +for all practical purposes. We then develop matrix perturbation tools that
 +quantify the objective, for embeddings explicitly or implicitly obtained from
 +matrix factorization. Practical, data-driven upper bounds will also be given.
 +Finally, we conduct extensive empirical studies and validate our theory on
 +real datasets. With this theory, we provide answers to three open questions
 +about vector embeddings, namely the robustness to over-parametrization,​
 +forward stability, and dimensionality selection.
 +https://​arxiv.org/​abs/​1802.03690v1 On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
 +Convolutional neural networks have been extremely successful in the image recognition domain because they ensure equivariance to translations. There have been many recent attempts to generalize this framework to other domains, including graphs and data lying on manifolds. In this paper we give a rigorous, theoretical treatment of convolution and equivariance in neural networks with respect to not just translations,​ but the action of any compact group. Our main result is to prove that (given some natural constraints) convolutional structure is not just a sufficient, but also a necessary condition for equivariance to the action of a compact group. Our exposition makes use of concepts from representation theory and noncommutative harmonic analysis and derives new generalized convolution formulae.
 +https://​arxiv.org/​pdf/​1805.06595.pdf Covariance-Insured Screening
 +existing screening methods, which typically ignore correlation information,​ are
 +likely to miss these weak signals. By incorporating the inter-feature dependence, we
 +propose a covariance-insured screening methodology to identify predictors that are
 +jointly informative but only marginally weakly associated with outcomes.
 +http://​www.uvm.edu/​~cdanfort/​courses/​237/​schmidt-lipson-2009.pdf Distilling Free-Form Natural Laws
 +from Experimental Data
 +https://​arxiv.org/​abs/​1805.12491 Structure from noise: Mental errors yield abstract representations of events
 +https://​arxiv.org/​abs/​1807.04689 Explorations in Homeomorphic Variational Auto-Encoding
 +In this paper we investigate the use of manifold-valued latent variables. Specifically,​ we focus on the important case of continuously differentiable symmetry groups (Lie groups), such as the group of 3D rotations SO(3). We show how a VAE with SO(3)-valued latent variables can be constructed,​ by extending the reparameterization trick to compact connected Lie groups. Our experiments show that choosing manifold-valued latent variables that match the topology of the latent data manifold, is crucial to preserve the topological structure and learn a well-behaved latent space.
 +https://​arxiv.org/​abs/​1808.05563 Learning Invariances using the Marginal Likelihood
 +We argue that invariances should instead be incorporated in the model structure, and learned using the marginal likelihood, which correctly rewards the reduced complexity of invariant models.
 +https://​arxiv.org/​abs/​1706.01350 Emergence of Invariance and Disentanglement in Deep Representations
 +We propose regularizing the loss by bounding such a term in two equivalent ways: One with a Kullbach-Leibler term, which relates to a PAC-Bayes perspective;​ the other using the information in the weights as a measure of complexity of a learned model, yielding a novel Information Bottleneck for the weights. Finally, we show that invariance and independence of the components of the representation learned by the network are bounded above and below by the information in the weights, and therefore are implicitly optimized during training. The theory enables us to quantify and predict sharp phase transitions between underfitting and overfitting of random labels when using our regularized loss, which we verify in experiments,​ and sheds light on the relation between the geometry of the loss function, invariance properties of the learned representation,​ and generalization error.
 +https://​arxiv.org/​pdf/​1809.02601v1.pdf Accelerating Deep Neural Networks with Spatial Bottleneck Modules
 +This paper presents an efficient module named spatial
 +bottleneck for accelerating the convolutional layers in deep
 +neural networks. The core idea is to decompose convolution
 +into two stages, which first reduce the spatial resolution
 +of the feature map, and then restore it to the desired size.
 +This operation decreases the sampling density in the spatial
 +domain, which is independent yet complementary to
 +network acceleration approaches in the channel domain.
 +Using different sampling rates, we can tradeoff between
 +recognition accuracy and model complexity
 +https://​arxiv.org/​abs/​1809.02591v1 Learning Invariances for Policy Generalization
 +While recent progress has spawned very powerful machine learning systems, those agents remain extremely specialized and fail to transfer the knowledge they gain to similar yet unseen tasks. In this paper, we study a simple reinforcement learning problem and focus on learning policies that encode the proper invariances for generalization to different settings. We evaluate three potential methods for policy generalization:​ data augmentation,​ meta-learning and adversarial training. We find our data augmentation method to be effective, and study the potential of meta-learning and adversarial learning as alternative task-agnostic approaches. ​
 +https://​openreview.net/​forum?​id=Ske25sC9FQ Robustness and Equivariance of Neural Networks
 +Robustness to rotations comes at the cost of robustness of pixel-wise adversarial perturbations.
 +https://​arxiv.org/​abs/​1809.10083v1 Unsupervised Adversarial Invariance
 +We present a novel unsupervised invariance induction framework for neural networks that learns a split representation of data through competitive training between the prediction task and a reconstruction task coupled with disentanglement,​ without needing any labeled information about nuisance factors or domain knowledge. We describe an adversarial instantiation of this framework and provide analysis of its working. Our unsupervised model outperforms state-of-the-art methods, which are supervised, at inducing invariance to inherent nuisance factors, effectively using synthetic data augmentation to learn invariance, and domain adaptation. Our method can be applied to any prediction task, eg., binary/​multi-class classification or regression, without loss of generality.
 +disentanglement is achieved between e1 and e2 in a novel way through two adversarial disentanglers
 +— one that aims to predict e2 from e1 and another that does the inverse.
 +https://​openreview.net/​forum?​id=BklHpjCqKm Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning ​