Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
random_matrix [2016/12/28 21:09]
127.0.0.1 external edit
random_matrix [2018/11/06 11:00] (current)
admin
Line 94: Line 94:
 training. The eigenvalue distribution is seen to be composed of two parts, the training. The eigenvalue distribution is seen to be composed of two parts, the
 bulk which is concentrated around zero, and the edges which are scattered away bulk which is concentrated around zero, and the edges which are scattered away
-from zero. We present empirical evidence for the bulk indicating how overparametrized+from zero. We present empirical evidence for the bulk indicating how over parametrized
 the system is, and for the edges indicating the complexity of the the system is, and for the edges indicating the complexity of the
 input data. input data.
Line 106: Line 106:
 https://​arxiv.org/​abs/​1608.05391 https://​arxiv.org/​abs/​1608.05391
  
 +https://​arxiv.org/​pdf/​1711.04735.pdf Resurrecting the sigmoid in deep learning through
 +dynamical isometry: theory and practice
  
 +https://​openreview.net/​pdf?​id=SJeFNoRcFQ TRADITIONAL AND HEAVY TAILED SELF REGULARIZATION
 +IN NEURAL NETWORK MODELS . https://​www.youtube.com/​watch?​v=_Ni5UDrVwYU
 +
 +https://​arxiv.org/​abs/​1810.01075v1 Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
 +
 +
 +https://​arxiv.org/​abs/​1805.11917v2 The Dynamics of Learning: A Random Matrix Approach
 +
 +we introduce a random matrix-based framework to analyze the learning dynamics of a single-layer linear network on a binary classification problem, for data of simultaneously large dimension and size, trained by gradient descent. Our results provide rich insights into common questions in neural nets, such as overfitting,​ early stopping and the initialization of training, thereby opening the door for future studies of more elaborate structures and models appearing in today'​s neural networks.
 +
 +https://​papers.nips.cc/​paper/​6857-nonlinear-random-matrix-theory-for-deep-learning.pdf Nonlinear random matrix theory for deep learning
 +
 +https://​stats385.github.io/​assets/​lectures/​Understanding_and_improving_deep_learing_with_random_matrix_theory.pdf