Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Last revision Both sides next revision
tensor_network [2017/08/15 23:42]
127.0.0.1 external edit
tensor_network [2017/11/03 19:44]
admin
Line 19: Line 19:
  
 To date, most convolutional neural network architectures output predictions by flattening 3rd-order activation tensors, and applying fully-connected output layers. This approach has two drawbacks: (i) we lose rich, multi-modal structure during the flattening process and (ii) fully-connected layers require many parameters. We present the first attempt to circumvent these issues by expressing the output of a neural network directly as the the result of a multi-linear mapping from an activation tensor to the output. By imposing low-rank constraints on the regression tensor, we can efficiently solve problems for which existing solutions are badly parametrized. Our proposed tensor regression layer replaces flattening operations and fully-connected layers by leveraging multi-modal structure in the data and expressing the regression weights via a low rank tensor decomposition. Additionally,​ we combine tensor regression with tensor contraction to further increase efficiency. Augmenting the VGG and ResNet architectures,​ we demonstrate large reductions in the number of parameters with negligible impact on performance on the ImageNet dataset. To date, most convolutional neural network architectures output predictions by flattening 3rd-order activation tensors, and applying fully-connected output layers. This approach has two drawbacks: (i) we lose rich, multi-modal structure during the flattening process and (ii) fully-connected layers require many parameters. We present the first attempt to circumvent these issues by expressing the output of a neural network directly as the the result of a multi-linear mapping from an activation tensor to the output. By imposing low-rank constraints on the regression tensor, we can efficiently solve problems for which existing solutions are badly parametrized. Our proposed tensor regression layer replaces flattening operations and fully-connected layers by leveraging multi-modal structure in the data and expressing the regression weights via a low rank tensor decomposition. Additionally,​ we combine tensor regression with tensor contraction to further increase efficiency. Augmenting the VGG and ResNet architectures,​ we demonstrate large reductions in the number of parameters with negligible impact on performance on the ImageNet dataset.
 +
 +https://​arxiv.org/​pdf/​1710.10248v2.pdf TENSOR NETWORK LANGUAGE MODEL
 +