Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
risk_minimization [2018/05/23 02:21]
admin
risk_minimization [2018/12/02 14:04] (current)
admin
Line 207: Line 207:
  
 We show explicitly for supervised learning of Boolean functions that the intrinsic simplicity bias of deep neural networks means that they generalize significantly better than an unbiased learning algorithm does.  We show explicitly for supervised learning of Boolean functions that the intrinsic simplicity bias of deep neural networks means that they generalize significantly better than an unbiased learning algorithm does. 
 +
 +https://​arxiv.org/​abs/​1708.06019 A Capacity Scaling Law for Artificial Neural Networks
 +
 +First, we derive the calculation of what we call the lossless memory (LM) dimension. The LM dimension is a generalization of the Vapnik-Chervonenkis (VC) dimension that avoids structured data and therefore provides an upper bound for perfectly fitting any training data. Second, we derive what we call the MacKay (MK) dimension. This limit indicates necessary forgetting, that is, the lower limit for most generalization uses of the network. ​
 +
 +https://​thegradient.pub/​frontiers-of-generalization-in-natural-language-processing/ ​
 +
 +https://​arxiv.org/​abs/​1808.08750 Generalisation in humans and deep neural networks
 +
 +We compare the robustness of humans and current convolutional deep neural networks (DNNs) on object recognition under twelve different types of image degradations. First, using three well known DNNs (ResNet-152,​ VGG-19, GoogLeNet) we find the human visual system to be more robust to nearly all of the tested image manipulations,​ and we observe progressively diverging classification error-patterns between humans and DNNs when the signal gets weaker. Secondly, we show that DNNs trained directly on distorted images consistently surpass human performance on the exact distortion types they were trained on, yet they display extremely poor generalisation abilities when tested on other distortion types. For example, training on salt-and-pepper noise does not imply robustness on uniform white noise and vice versa. Thus, changes in the noise distribution between training and testing constitutes a crucial challenge to deep learning vision systems that can be systematically addressed in a lifelong machine learning approach. Our new dataset consisting of 83K carefully measured human psychophysical trials provide a useful reference for lifelong robustness against image degradations set by the human visual system.
 +
 +https://​arxiv.org/​abs/​1802.08598 Learning Weighted Representations for Generalization Across Designs
 +
 +https://​arxiv.org/​abs/​1703.02660v2 Towards Generalization and Simplicity in Continuous Control
 +
 +