Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
mutual_information [2018/03/16 21:31]
admin
mutual_information [2018/10/02 10:19] (current)
admin
Line 366: Line 366:
 Finally, we suggest that such decompositions could aid the design of context-sensitive Finally, we suggest that such decompositions could aid the design of context-sensitive
 machine learning algorithms machine learning algorithms
 +
 +https://​arxiv.org/​abs/​1801.09223v2 Probability Mass Exclusions and the Directed Components of Pointwise Mutual Information
 +
 +We start by introducing probability mass diagrams, which provide a visual representation of how a prior distribution is transformed to a posterior distribution through exclusions. With the aid of these diagrams, we identify two distinct types of probability mass exclusions---namely informative and misinformative exclusions.
 +
 +https://​arxiv.org/​abs/​1805.04928v1 Doing the impossible: Why neural networks can be trained at all
 +
 +In the current work we use the concept of mutual information between successive layers of a deep neural network to elucidate this mechanism and suggest possible ways of exploiting it to accelerate training. We show that adding structure to the neural network that enforces higher mutual information between layers speeds training and leads to more accurate results. High mutual information between layers implies that the effective number of free parameters is exponentially smaller than the raw number of tunable weights.
 +
 +https://​arxiv.org/​abs/​1805.07249 Dynamic learning rate using Mutual Information
 +
 +Two approaches are demonstrated - tracking relative change in mutual information and, additionally tracking its value relative to a reference measure.
 +
 +https://​arxiv.org/​abs/​1808.06670v1 Learning deep representations by mutual information estimation and maximization
 +
 +Our method, which we call Deep INFOMAX (DIM), can be used to learn representations with desired characteristics and which empirically outperform a number of popular unsupervised learning methods on classification tasks. DIM opens new avenues for unsupervised learn-ing of representations and is an important step towards flexible formulations of representation learning objectives catered towards specific end-goals.
 +
 +https://​xbpeng.github.io/​projects/​VDB/​index.html Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow