Deep Learning Patterns
http://www.deeplearningpatterns.com/
2018-02-21T10:56:45+00:00Deep Learning Patterns
http://www.deeplearningpatterns.com/
http://www.deeplearningpatterns.com/lib/tpl/bootstrap3/images/favicon.icotext/html2018-02-20T10:40:53+00:00admincurriculum_training
http://www.deeplearningpatterns.com/doku.php?id=curriculum_training&rev=1519123253&do=diff
Curriculum Training
Aliases
Intent
Train the network with the easiest examples first and gradually increasing the difficulty.
Motivation
How can we speed up training?
Sketch
This section provides alternative descriptions of the pattern in the form of an illustration or alternative formal expression. By looking at the sketch a reader may quickly understand the essence of the pattern.text/html2018-02-19T13:02:18+00:00adminhyper-parameter_tuning
http://www.deeplearningpatterns.com/doku.php?id=hyper-parameter_tuning&rev=1519045338&do=diff
.Hyper Parameter Tuning
Known Uses
<https://code.facebook.com/posts/1072626246134461/introducing-fblearner-flow-facebook-s-ai-backbone/>
Machine learning automation: Many machine learning algorithms have numerous hyperparameters that can be optimized. At Facebook's scale, a 1 percent improvement in accuracy for many models can have a meaningful impact on people's experiences. So with Flow, we built support for large-scale parameter sweeps and other AutoML features that leverage idle cycles to …text/html2018-02-19T12:59:28+00:00adminensembles
http://www.deeplearningpatterns.com/doku.php?id=ensembles&rev=1519045168&do=diff
Edit: <https://docs.google.com/a/codeaudit.com/document/d/1xD1CGw79a_1yipq7cxhnBe1a91R9nwGA4XuXdRU7tHA/edit?usp=sharing>
Ensemble
Aliases Mixture of Experts
Intent
A more accurate network can be created using a combination of existing individually trained networks.
Motivation
How can we achieve higher predictive accuracy by combining independently trained networks?text/html2018-02-19T01:01:57+00:00adminlearning_to_purpose
http://www.deeplearningpatterns.com/doku.php?id=learning_to_purpose&rev=1519002117&do=diff
Inverse Learning
Aliases Cost Function Learning
<https://phillipi.github.io/pix2pix/> Image-to-Image Translation with Conditional Adversarial Nets
We investigate conditional adversarial networks as a
general-purpose solution to image-to-image translation
problems. These networks not only learn the mapping from
input image to output image, but also learn a loss function
to train this mapping.text/html2018-02-17T04:04:33+00:00admintutorials
http://www.deeplearningpatterns.com/doku.php?id=tutorials&rev=1518840273&do=diff
<http://beamandrew.github.io/deeplearning/2017/02/23/deep_learning_101_part1.html>
<https://github.com/martin-gorner/tensorflow-mnist-tutorial>
<https://github.com/aymericdamien/TensorFlow-Examples>
<https://github.com/hpssjellis/easy-tensorflow-on-cloud9>
<https://github.com/nlintz/TensorFlow-Tutorials>
<https://github.com/Hvass-Labs/TensorFlow-Tutorials>
<https://github.com/LeavesBreathe/tensorflow_with_latest_papers>
<https://github.com/sjchoi86/Tensorflow-101>
<http://www.thoughtly.c…text/html2018-02-16T20:21:53+00:00adminmodularity
http://www.deeplearningpatterns.com/doku.php?id=modularity&rev=1518812513&do=diff
Edit: <https://drive.google.com/open?id=1TjrqVuYQNFmBlpUNoThI53j49SSTMr_VZ6RCQNQnuxM>
Modularity
Aliases
This identifies the pattern and should be representative of the concept that it describes. The name should be a noun that should be easily usable within a sentence. We would like the pattern to be easily referenceable in conversation between practitioners.text/html2018-02-16T20:15:52+00:00admintransport_related
http://www.deeplearningpatterns.com/doku.php?id=transport_related&rev=1518812152&do=diff
<https://arxiv.org/pdf/1609.04767v1.pdf> Transport-based analysis, modeling, and learning
from signal and data distributions
the geometric characteristics of
transport-related metrics have inspired new kinds of algorithms
for interpreting the meaning of data distributions. Here we
provide an overview of the mathematical underpinnings of mass
transport-related methods, including numerical implementation,
as well as a review, with demonstrations, of several applications.text/html2018-02-16T14:08:47+00:00adminrationalization
http://www.deeplearningpatterns.com/doku.php?id=rationalization&rev=1518790127&do=diff
<https://arxiv.org/abs/1606.04155> Rationalizing Neural Predictions
Prediction without justification has limited applicability. As a remedy, we learn to extract pieces of input text as justifications -- rationales -- that are tailored to be short and coherent, yet sufficient for making the same prediction.text/html2018-02-16T13:33:39+00:00adminprobabilistic_graph
http://www.deeplearningpatterns.com/doku.php?id=probabilistic_graph&rev=1518788019&do=diff
Probabilistic Graph Model Integration
<https://arxiv.org/pdf/1602.06822.pdf> Understanding Visual Concepts with Continuation Learning
We introduce a neural network architecture and a learning algorithm to produce factorized symbolic representations. We propose to learn these concepts by observing consecutive frames, letting all the components of the hidden representation except a small discrete set (gating units) be predicted from the previous frame, and let the factors of variation in the nex…text/html2018-02-16T11:43:31+00:00adminapplications
http://www.deeplearningpatterns.com/doku.php?id=applications&rev=1518781411&do=diff
Applications
When we first are introduced to deep learning, we see it as a better machine learning classifier. Alternatively, we could subscribe to the hype that it is 'brain-like' neuro-computing. In the former instance, we grossly underestimate the kinds of applications we can build with this. In the later instance, we grossly overestimate its capabilities and as a consequence overlook the kind of applications that are not general artificial intelligence, but applications that are more r…text/html2018-02-16T11:35:01+00:00admincontext
http://www.deeplearningpatterns.com/doku.php?id=context&rev=1518780901&do=diff
<http://vision.princeton.edu/projects/2016/DeepContext/paper.pdf> DeepContext: Context-Encoding Neural
Pathways for 3D Holistic Scene Understanding <https://arxiv.org/pdf/1603.04922v3.pdf>
In particular, 3D context has been
shown to be an extremely important cue for scene understanding - yet
very little research has been done on integrating context information
with deep models. This paper presents an approach to embed 3D context
into the topology of a neural network trained to perform holisti…text/html2018-02-16T11:34:33+00:00adminneural_embedding
http://www.deeplearningpatterns.com/doku.php?id=neural_embedding&rev=1518780873&do=diff
Edit: <https://docs.google.com/a/codeaudit.com/document/d/1xdZkMtYH_NuY6Rk2bKWOLwAgwyilY8YzzYY0sZ8BMXs/edit?usp=sharing>
Name Neural Embedding (aka Vectorization, Word2Vec, *2Vec)
Intent Analogy through algebra.
Complex features can be projected into lower dimensions while capture intrinsic semantics.
Motivation
Complex features can exists at extremely high dimensions and thus requiring an unbounded amount of computational resources to perform classification.text/html2018-02-16T01:49:56+00:00admindiscrete_model
http://www.deeplearningpatterns.com/doku.php?id=discrete_model&rev=1518745796&do=diff
Also: Conditional Computation
<https://arxiv.org/abs/1308.3432> Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
Stochastic neurons and hard non-linearities can be useful for a number of reasons in deep learning models, but in many cases they pose a challenging problem: how to estimate the gradient of a loss function with respect to the input of such stochastic or non-smooth neurons? I.e., can wetext/html2018-02-15T19:31:48+00:00adminiterative_inference
http://www.deeplearningpatterns.com/doku.php?id=iterative_inference&rev=1518723108&do=diff
<https://arxiv.org/pdf/1707.09219.pdf> Recurrent Ladder Networks
We propose a recurrent extension of the Ladder network [24], which is motivated
by the inference required in hierarchical latent variable models. We demonstrate
that the recurrent Ladder is able to handle a wide variety of complex learning
tasks that benefit from iterative inference and temporal modeling.text/html2018-02-15T19:04:14+00:00adminexplanatory
http://www.deeplearningpatterns.com/doku.php?id=explanatory&rev=1518721454&do=diff
<https://docs.google.com/a/codeaudit.com/document/d/18fymZOPiTuil989vV3aNFuLVD3I4vG53TPeDJ6_j6G8/edit?usp=sharing>
Explanatory Patterns
Note: This chapter is undergoing massive refactoring
Neural networks are originally designed to perform classification well. Furthermore, these networks perform in a restricted domain (i.e. vision and speech recognition). Researchers have also been conveniently able to perform comparative benchmarks between their proposed research and earlier research. …text/html2018-02-15T18:50:47+00:00adminactive_learning
http://www.deeplearningpatterns.com/doku.php?id=active_learning&rev=1518720647&do=diff
<https://arxiv.org/abs/1701.03551> Cost-Effective Active Learning for Deep Image Classification
Through the properly designed framework, the feature representation and the classifier can be simultaneously updated with progressively annotated informative samples. Second, we present a cost-effective sample selection strategy to improve the classification performance with less manual annotations.text/html2018-02-15T18:31:38+00:00adminmanifold_traversal
http://www.deeplearningpatterns.com/doku.php?id=manifold_traversal&rev=1518719498&do=diff
Manifold Traversal
Aliases
Intent
Train the network so that there is a continuous manifold that can be navigated as an aid in explanation.
Motivation
How can we traverse the input space in a way that explains the outputs?
Sketch
This section provides alternative descriptions of the pattern in the form of an illustration or alternative formal expression. By looking at the sketch a reader may quickly understand the essence of the pattern.text/html2018-02-15T17:34:43+00:00adminnatural_gradient_descent
http://www.deeplearningpatterns.com/doku.php?id=natural_gradient_descent&rev=1518716083&do=diff
Discussion
The dynamics of a Neural Network is usually framed in the context of optimizing a convex or non-convex non linear problem. This involves the minimization/maximization of an objective function. The formulation of the objective function is a bit arbitrary but it is typically the squared error between the actual and estimated values:text/html2018-02-15T03:15:48+00:00adminweight_quantization
http://www.deeplearningpatterns.com/doku.php?id=weight_quantization&rev=1518664548&do=diff
Name
Weight Quantization (aka Binarization)
Intent
Reduce memory requirements by using weights of lower precision.
Motivation
Structure
<Diagram>
Discussion
Known Uses
Related Patterns
<Diagram>
References
<http://arxiv.org/pdf/1606.00185v1.pdf> A Survey on Learning to Hashtext/html2018-02-15T03:09:04+00:00adminrandom_projections
http://www.deeplearningpatterns.com/doku.php?id=random_projections&rev=1518664144&do=diff
Edit: <https://drive.google.com/open?id=1G3-MoDSI7cN3xO-ujdYZoeVwuYww-A_Hm7NHiInSOpU>
Random Projections
Aliases SimHash
Intent
Use random projections to construct a mapping that preserves locality.
Motivation
How to create an acceptable mapping between spaces while preserving locality of similar objects?
Structure
<Diagram>