Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
random_orthogonal_initialization [2017/07/16 19:44]
127.0.0.1 external edit
random_orthogonal_initialization [2018/11/04 15:14] (current)
admin
Line 351: Line 351:
  
 https://​r2rt.com/​non-zero-initial-states-for-recurrent-neural-networks.html https://​r2rt.com/​non-zero-initial-states-for-recurrent-neural-networks.html
 +
 +https://​severelytheoretical.wordpress.com/​2018/​01/​01/​why-is-it-hard-to-train-deep-neural-networks-degeneracy-not-vanishing-gradients-is-the-key/​
 +
 +https://​arxiv.org/​abs/​1802.09979v1 The Emergence of Spectral Universality in Deep Networks
 +
 +Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network'​s input-output Jacobian around one at initialization can speed up learning by orders of magnitude.
 +
 +Our results provide a
 +principled framework for the initialization of weights
 +and the choice of nonlinearities in order to produce
 +well-conditioned Jacobians and fast learning. Intriguingly,​
 +we find novel universality classes of deep spectra
 +that remain well-conditioned as the depth goes to
 +infinity, as well as theoretical conditions for their existence.
 +
 +https://​arxiv.org/​abs/​1804.03758 Universal Successor Representations for Transfer Reinforcement Learning
 +
 +The objective of transfer reinforcement learning is to generalize from a set of previous tasks to unseen new tasks. In this work, we focus on the transfer scenario where the dynamics among tasks are the same, but their goals differ. Although general value function (Sutton et al., 2011) has been shown to be useful for knowledge transfer, learning a universal value function can be challenging in practice. To attack this, we propose (1) to use universal successor representations (USR) to represent the transferable knowledge and (2) a USR approximator (USRA) that can be trained by interacting with the environment. Our experiments show that USR can be effectively applied to new tasks, and the agent initialized by the trained USRA can achieve the goal considerably faster than random initialization.
 +
 +https://​arxiv.org/​abs/​1806.10909 ResNet with one-neuron hidden layers is a Universal Approximator
 +
 +https://​arxiv.org/​pdf/​1809.08836.pdf Dense neural networks as sparse graphs and the
 +lightning initialization
 +
 +