深度学习资料：Learning Deep Learning

贾伟中科院合肥

http://rt.dgyblog.com/ref/ref-learning-deep-learning.html

Learning Deep Learning

[NOTICE] I'm in the middle of adding NLP and Speech Recognition key references.

There are lots of awesome reading lists or posts that summarized materials related to Deep Learning. So why would I commit another one? Well, the primary objective is to develop a complete reading list that allows readers to build a solid academic and practical background of Deep Learning. And this list is developed while I'm preparing my Deep Learning workshop. My research is related to Deep Neural Networks (DNNs) in general. Hence, this posts tends to summary contributions in DNNs instead of generative models.

For Novice

If you have no idea about Machine Learning and Scientific Computing, I suggest you learn the following materials while you are reading Machine Learning or Deep Learning books. You don't have to master these materials, but basic understanding is important. It's hard to open a meaningful conversation if the person has no idea about matrix or single variable calculus.

• Introduction to Algorithms by Erik Demaine and Srinivas Devadas.
• Single Variable Calculus by David Jerison.
• Multivariable Calculus by Denis Auroux.
• Differential Equations by Arthur Mattuck, Haynes Miller, Jeremy Orloff, John Lewis.
• Linear Algebra by Gilbert Strang.
  

Theory of Computation, Learning Theory, Neuroscience, etc

• Introduction to the Theory of Computation by Michael Sipser.
• Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig.
• Pattern Recognition and Machine Learning by Christopher Bishop.
• Machine Learning: A probabilistic perspective by Kevin Patrick Murphy.
• CS229 Machine Learning Course Materials by Andrew Ng at Stanford University.
• Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto.
• Probabilistic Graphical Models: Principles and Techniques by Daphne Koller and Nir Friedman.
• Convex Optimization by Stephen Boyd and Lieven Vandenberghe.
• An Introduction to Statistical Learning with application in R by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani.
• Neuronal Dynamics: From single neurons to networks and models of cognition by Wulfram Gerstner, Werner M. Kistler, Richard Naud and Liam Paninski.
• Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems by Peter Dayan and Laurence F. Abbott.
• Michael I. Jordan Reading List of Machine Learning at Hacker News.
  
  

Fundamentals of Deep Learning

• Deep Learning in Neural Networks: An Overview by Jürgen Schmidhuber.
• Deep Learning Book by Yoshua Bengio, Ian Goodfellow and Aaron Courville.
• Learning Deep Architectures for AI by Yoshua Bengio
• Representation Learning: A Review and New Perspectives by Yoshua Bengio, Aaron Courville, Pascal Vincent.
  
  

Tutorials, Practical Guides, and Useful Software

• Machine Learning by Andrew Ng.
• Neural Networks for Machine Learning by Geoffrey Hinton.
• Deep Learning Tutorial by LISA Lab, University of Montreal.
• Unsupervised Feature Learning and Deep Learning Tutorial by AI Lab, Stanford University.
• CS231n: Convolutional Neural Networks for Visual Recognition by Stanford University.
• CS224d: Deep Learning for Natural Language Processing by Stanford University.
• Theano by LISA Lab, University of Montreal.
• PyLearn2 by LISA Lab, University of Montreal.
• Caffe by Berkeley Vision and Learning Center (BVLC) and community contributor Yangqing Jia.
• Torch 7
• neon by Nervana.
• cuDNN by NVIDIA.
• ConvNetJS by Andrej Karpathy.
• DeepLearning4j
  
  

Literature in Deep Learning and Feature Learning

Deep Learning is a fast-moving community. Therefore the line between "Recent Advances" and "Literature that matter" is kind of blurred. Here I collected articles that are either introducing fundamental algorithms, techniques or highly cited by the community.

• Backpropagation Applied to Handwritten Zip Code Recognition by Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard and L. D. Jackel.
• Comparison of Training Methods for Deep Neural Networks by Patrick O. Glauner.
• Distributed Representations of Words and Phrases and their Compositionality by Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado and Jeffrey Dean.
• Efficient Estimation of Word Representations in Vector Space by Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean.
• From Frequency to Meaning: Vector Space Models of Semantics by Peter D. Turney and Patrick Pantel.
• LSTM: A Search Space Odyssey by Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, Jürgen Schmidhuber.
• Supervised Sequence Labelling with Recurrent Neural Networks by Alex Graves.
  
  

Recent Must-Read Advances in Deep Learning

Most of papers here are produced in 2014 and after. Survey papers or review papers are not included.

• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images by Mateusz Malinowski, Marcus Rohrbach, Mario Fritz.
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift by Sergey Ioffe and Christian Szegedy.
• Beyond Short Snipets: Deep Networks for Video Classification by Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, George Toderici.
• DeepBox: Learning Objectness with Convolutional Networks by Weicheng Kuo, Bharath Hariharan, Jitendra Malik.
• Deep Networks with Internal Selective Attention through Feedback Connections by Marijn Stollenga, Jonathan Masci, Faustino Gomez, Jürgen Schmidhuber.
• Deep Visual-Semantic Alignments for Generating Image Descriptions by Andrej Karpathy, Fei-Fei Li.
• Describing Videos by Exploiting Temporal Structure by Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville.
• DRAW: A Recurrent Neural Network For Image Generation by Karol Gregor, Ivo Danihelka, Alex Graves, Daan Wierstra.
• Dropout: A Simple Way to Prevent Neural Networks from Overfitting by Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov.
• FaceNet: A Unified Embedding for Face Recognition and Clustering by Florian Schroff, Dmitry Kalenichenko, James Philbin.
• Fast-Classifying, High-Accuracy Spiking Deep Networks Through Weight and Threshold Balancing by Peter U. Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer.
• FitNets: Hints for Thin Deep Nets by Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio.
• FlowNet: Learning Optical Flow with Convolutional Networks by Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazırbaş, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, Thomas Brox.
• Human-level control through deep reinforcement learning by Google DeepMind.
• ImageNet Classification with Deep Convolutional Neural Networks by Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton.
• Image Question Answering: A Visual Semantic Embedding Model and a New Dataset by Mengye Ren, Ryan Kiros, Richard Zemel.
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images by Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, Alan Yuille.
• Long-term Recurrent Convolutional Networks for Visual Recognition and Description by Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, Trevor Darrell.
• Maxout Networks by Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, Yoshua Bengio.
• Recurrent Models of Visual Attention by Volodymyr Mnih, Nicolas Heess, Alex Graves, Koray Kavukcuoglu.
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks by Francesco Visin, Kyle Kastner, Kyunghyun Cho, Matteo Matteucci, Aaron Courville, Yoshua Bengio.
• Searching for Higgs Boson Decay Modes with Deep Learning by Peter Sadowski, Pierre Baldi, Daniel Whiteson.
• Show and Tell: A Neural Image Caption Generator by Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention by Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio.
• Towards Biologically Plausible Deep Learning by Yoshua Bengio, Dong-Hyun Lee, Jorg Bornschein, Zhouhan Lin.
• Training Deeper Convolutional Networks with Deep Supervision by Liwei Wang, Chen-Yu Lee, Zhuowen Tu, Svetlana Lazebnik.
• Translating Videos to Natural Language Using Deep Recurrent Neural Networks by Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, Kate Saenko.
• Unsupervised Learning of Video Representations using LSTMs by Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov.
• Unsupervised Learning of Visual Representations using Videos by Xiaolong Wang, Abhinav Gupta.
• Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research by Atousa Torabi, Christopher Pal, Hugo Larochelle, Aaron Courville.
  
  

Podcast, Talks, etc.

• Talking Machines hosted by Katherine Gorman and Ryan Adams.
• How we're teaching computers to understand pictures by Fei-Fei Li, Stanford University.
• Deep Learning Community
  
  

Amazon Web Service Public AMI for Deep Learning

I configured 2 types of GPU instances that are available in AWS and installed necessary software for Deep Learning practice. The first one is DGYDLGPUv4. It is a machine that provides 8-core CPU, 15GB RAM, 500GB SSD and 1 NVIDIA Grid K520 GPU, you can use it to learn Deep Learning or conduct normal size experiment. If you need even more computing resources, you can choose DGYDLGPUXv1. This new released GPU instance offered a 32-core CPU, 60GB RAM, 500GB SSD and 4 NVIDIA Grid K520 GPUs.

NVIDIA Driver, CUDA Toolkit 7.0, cuDNNv2, Anaconda, Theano are preinstalled.

Currently they are only available at Asia Pacific (Singapore). You can copy the instance to your close region.

If you are doing analysis or experiments, I suggest you to request spot instance instead of on-demand instance. This will save you a lot of costs.

• DGYDLGPUv4 (ami-ba516ee8) [Based on g2.2xlarge]
• DGYDLGPUXv1 (ami-52516e00) [Based on g2.8xlarge]
  

Currently, my build of Caffe in the instance is failed. You can use the AMI that is provided by Caffe community. You can get more details from here

So far the instance is only available at US East (N. Virginia)

• Caffe/CuDNN built 2015-05-04 (ami-763a331e) [For both g2.2xlarge and g2.8xlarge]
  

Practical Deep Neural Networks - GPU computing perspective

The following entries are materials I use in the workshop.

Slides

• Introduction
• Python Platform for Scientific Computing
• Theano Crash Course
• [Machine Learning Basics]
• [Support Vector Machine and Softmax Regression]
• [Feedforward Neural Networks]
• [Convolutional Neural Networks]
• [Recurrent Neural Networks]
  

Practical tutorials

• Python Warm-up, pre-processing
• Support Vector Machine, Softmax Regression
• Multi Layer Perceptron Network
• Auto-encoder
• Convolutional Neural Networks
• Recurrent Neural Networks
  

Codes

• Telauges
  
  • A new deep learning libraries for education
  • Right now implements MLP Hidden Layer, Softmax Layer, Auto-encoder, ConvNet Layer
  • In the middle of transforming coding style, I suggest you explore it, but do not use it seriously.
    

徐国智华电

很好的资料，感谢分享！

莫娟四川大学

资料很全，总结的很好，多谢分享！