Fall 2018 - CIS 5590

 

Introduction to Deep Learning

 


Basic Information:

·  Lecture time: Wednesday 5:30-8:00pm, TTLMAN 302

·  Instructor: Haibin Ling | SERC 382 | 215-204-6973 | hbling AT temple.edu

·  Office Hours: Wednesday 3:00-5:00pm, or by appointment

·  Syllabus: PDF

Links
Temple Univ.
CIS Dept.
DABI Center

 

Important Linkes:

· FAQs and announcements: general FAQ, about review, about presentation.

· Main textbook (referred as GBC): Deep Learning, Ian Goodfellow, Yoshua Bengio, Aaron Courville, 2016, MIT press.

· Reference material: Deep learning with Python, François Chollet, Manning Publications; 1st edition, 2017.

· Reference material: Stanford deep learning course: CS231n: Convolutional Neural Networks for Visual Recognition.

· Reference material: Papers assigned in the class.

 


Date

Topics and Materials (Tentative, will be revised FREQUENTLY)

Additional References

Week 1
8/29

General introduction

· History and background introduction.

· Course introduction.

· Logistics

 

Quiz 1 (FAQ)

 

Materials

· Chapter 1 of GBC (the main textbook)

 

Paper presentation (sample, no review required)

· [Presenter: Peng Chu] Deep Learning of Graph Matching, Zanfir & Sminchisescu, CVPR 2018.

· Deep learning. LeCun, Bengio, & Hinton, Nature 2015.

· Deep neural networks for acoustic modeling in speech recognition. Hinton, et al., IEEE Signal Processing Magazine  2012.

· Relational inductive biases, deep learning, and graph networks. Bataglia et al. 2018.

 

 

 

Week 2
9/5

Preliminary

· Related basic knowledge in optimization and information theory.

· Deep learning in computer vision.

 

Materials

· Chapters 2-5 of GBC

· Related chapters in Computer Vision: Algorithms and Applications, Richard Szeliski, 2010.

· Convex optimization by Boyd & Vandenberghe, 2004.

 

 

9/10

 

Last day to drop the course (seriously)

 

Week 3
9/12

 

 

Deep Forward Networks

· Deep architecture

· Feed forward

· Training


Materials

· Chapters 6 of GBC

· Rumelhart, Hinton, & Williams. Learning representations by back-propagating errors. Nature 1986.

· Glorot, Bordes, & Bengio. Deep sparse rectifier neural networks. AISTAT 2011.

 

 

 

Week 4
9/19

 

 

 

Regularization for Deep Learning

· Various regularization techniques

.

Materials

· Chapters 7 of GBC

· Geometric deep learning: going beyond Euclidean data. Bronstein et al. IEEE SPM 2017.

 

 

 

 

Week 5
9/26

Optimization for Deep Learning

· Gradient descent and structure of cost functions

· Batch normalization

 

Materials

· Chapters 8 of GBC

 

Paper presentation (Review assignment)

· [Presenter: Heng Fan] Group Normalization. Wu & He. ECCV 2018

· [Presenter: Xinyi Li] On the Convergence of Adam and Beyond. Reddi, Kale, & Kumar. ICLR 2018.

· A fast learning algorithm for deep belief nets. Hinton, Osindero, & Teh. Neural Comp. 2006.

· Greedy layer-wise training of deep networks. Bengio, Lamblin, Popovici, & Larochelle. NIPS 2006.

 

 

 

 

Week 6
10/3

Convolutional Neural Networks

· Convolutional layers

· AlexNet

 

Materials

· Chapters 9 of GBC

· ImageNet Classification with Deep Convolutional Neural Networks, Krizhevsky & Hinton, NIPS 2012.

 

Paper presentation (Review assignment)

· [Presenter: Shikai Fang] Deep Convolutional Networks on Graph-Structured Data, Henaff, Bruna, & LeCun, 2015.

· [Presenter: Fengjiao Li] Semi-supervised Classification with Graph Convolutional Networks, Kipf & Welling. ICLR 2017

· LeCun, et al. Handwritten digit recognition with a back-propagation network. NIPS 1990.

· Gradient-based learning applied to document recognition. LeCun, Bottou, Bengio, & Haffner. Proc. IEEE 1998.

· Visualizing and Understanding Convolutional Networks, Zerler & Fergus, ECCV 2014.

 

 

 

Week 7
10/10

 Popular CNN Architectures

· VGGNet, GoogLeNet, ResNet

 

Materials

· Very Deep Convolutional Networks for Large-Scale Image Recognition. Simonyan & Zisserman, ICLR 2015.

· Going Deeper with Convolutions. Szegedy et al. CVPR 2015.

· Deep Residual Learning for Image Recognition. He, Zhang, Ren, & Sun, CVPR 2016.

 

Paper presentation (Review assignment)

· [Presenter: Anish Shah] SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size. Iandola et al. ICLR 2017.

· [Presenter: Zhongdong Liu] Aggregated Residual Transformations for Deep Neural Networks. Xie et al. CVPR 2017.

· [Presenter: Jumanah Alshehri] MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. Howard et al. 2017.

· Network In Network. Lin, Chen, & Yan. ICLR 2014.

· Densely Connected Convolutional Networks, Huang, Liu, van der Maaten, Weinberger, CVPR 2017.

 

 

 

 

Week 8
10/17

Popular Toolboxes

· Caffe, pyTorch, Tensorflow, MXNet

 

Materials

· Related papers

· TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems, Abadi et al., OSDI 2016.

· Caffe: Convolutional Architecture for Fast Feature Embedding, Jia et al., 2014

· PyTorch

· MxNet

 

Quiz 2

 

 

10/23

 

Last day to withdraw the course

 

Week 9
10/24

 

Sequence Modeling

· Recurrent Neural Networks (RNN)

· Long Short Term Memory (LSTM)

 

Materials

· Chapters 10 of GBC

 

Project proposal due

· Electronic version due before class (email to me before the class starts)

· Using ICLR 2018 template, download.

· Requirement:

-        Strictly following the ICLR template, including font and page sizes

-        Should contain at least the following sections: (1) Introduction, (2) Proposed research, (3) Evaluation plan, and (4) References

-        Minimum three pages, excluding the references.

 

Paper presentation (Review assignment)

· [Presenter: Saman Enayati] Sequence to sequence learning with neural networks. Sutskever, Vinyals, & Le. NIPS 2014.

· [Presenter: Nicholas Rork Torba] Neural Machine Translation by Jointly Learning to Align and Translate. Bahdanau, Cho, & Bengio. ICLR 2015.

· Recurrent Neural Networks, Wikipedia.

· Long Short-Term Memory. Hochreiter, Sepp; Schmidhuber, Jürgen. Neural Computation, 1997.

 

 

 

 

Week 10
10/31

Practical Methodology

· Practical issues in modeling, training and data augmentation.

 

Materials

· Chapters 11 of GBC

 

Paper presentation (Review assignment)

· [Presenter: Sheng Zhang] Attention is All you Need. Vaswani et al. NIPS 2017.

· [Presenter: Sidra Hanif] Dynamic Routing Between Capsules. Sabour, Frosst, & Hinton. NIPS 2017.

· Fully Convolutional Networks for Semantic Segmentation. J. Long, E. Shelhamer, T. Darrell, CVPR 2015.

 

 

 

 

Week 11
11/7

Applications

· Brief summarization of sampled applications

· Computer vision tasks (detection, tracking, segmentation)

 

Materials

· Chapters 12 of GBC

 

Paper presentation (Review assignment)

· [Presenter: Yanlong Qiu] Focal Loss for Dense Object Detection. Lin, Goyal, Girshick, He, & Dollár. ICCV 2017.

· [Presenter: Xuening Xu] Fully-Convolutional Siamese Networks for Object Tracking. Bertinetto et al.  ECCV Workshop 2016

 

· Deep Learning for generic object detection: a survey. Liu et al. ECCV 2018.

· Region-based convolutional networks for accurate object detection and segmentation, Girshick, Donahue, Darrell, Malik. CVPR 2014.

· YOLO9000: Better, Faster, Stronger. Redmon & Farhadi, CVPR, 2017.

 

 

 

Week 12
11/14

Unsupervised Deep Learning

· Autoencoders

· Generative Adversarial Networks

 

Materials

· Chapters 14 of GBC

 

Paper presentation (Review assignment)

· [Presenter: Patrick Hammer] Wasserstein Auto-Encoders, Tolstikhin, Bousquet, Gelly, & Schoelkopf. ICLR 2018.

· [Presenter: Bingyao Huang] Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. Ledig et al. CVPR 2017.

· Reducing the dimensionality of data with neural networks. Hinton & Salakhutdinov, Science 2006.

· Generative Adversarial Networks, Goodfellow et al., NIPS 2014.

 

 

 

 

Week 13
11/28

Advanced Topics (tentative)

· Deep reinforcement learning

 

Materials

· Chapters 12 of GBC

 

Paper presentation (Review assignment)

· [Presenter: Sandro Hauri] Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm. Silver et al. NIPS 2017.

· [Presenter: Terrel Nowlin] Neural Architecture Search with Reinforcement Learning. Zoph & Le. ICLR 2017

 

 

 

 

Week 14
12/5

Advanced Topics (tentative)

· Emerging topics

 

Project presentation

 

 

 

 

12/12

Project report due!