Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
residual [2018/03/09 22:48]
admin
residual [2018/04/23 11:26] (current)
admin
Line 168: Line 168:
 Resnets are able to perform both representation learning and iterative refinement. In general, a Resnet block tends to concentrate representation learning behavior in the first few layers while higher layers perform iterative refinement of features. Finally we observe that sharing residual layers naively leads to representation explosion and counterintuitively,​ overfitting,​ and we show that simple existing strategies can help alleviating this problem. Resnets are able to perform both representation learning and iterative refinement. In general, a Resnet block tends to concentrate representation learning behavior in the first few layers while higher layers perform iterative refinement of features. Finally we observe that sharing residual layers naively leads to representation explosion and counterintuitively,​ overfitting,​ and we show that simple existing strategies can help alleviating this problem.
  
-Unshared Batch Normalization strategy therefore mitigates this exploding activation problem. ​T+Unshared Batch Normalization strategy therefore mitigates this exploding activation problem. ​
  
 +https://​arxiv.org/​abs/​1804.07209 NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
 +
 +Non-autonomy is implemented
 +by skip connections from the block input
 +to each of the unrolled processing stages and allows
 +stability to be enforced so that blocks can
 +be unrolled adaptively to a pattern-dependent processing
 +depth. We prove that the network is globally
 +asymptotically stable so that for every initial
 +condition there is exactly one input-dependent
 +equilibrium assuming tanh units, and multiple
 +stable equilibria for ReL units.
 +
 +We believe that cross-breeding machine learning and control
 +theory will open up many new interesting avenues for
 +research, and that more robust and stable variants of commonly
 +used neural networks, both feed-forward and recurrent,
 +will be possible.