Fall 2018 - CIS 5590


Introduction to Deep Learning


Basic Information:

·  Lecture time: Wednesday 5:30-8:00pm, TTLMAN 302

·  Instructor: Haibin Ling | SERC 382 | 215-204-6973 | hbling AT temple.edu

·  Office Hours: Wednesday 3:00-5:00pm, or by appointment

·  Syllabus: PDF

Temple Univ.
CIS Dept.
DABI Center


Important Linkes:

· FAQs and announcements: general FAQ, about review, about presentation.

· Main textbook (referred as GBC): Deep Learning, Ian Goodfellow, Yoshua Bengio, Aaron Courville, 2016, MIT press.

· Reference material: Deep learning with Python, François Chollet, Manning Publications; 1st edition, 2017.

· Reference material: Stanford deep learning course: CS231n: Convolutional Neural Networks for Visual Recognition.

· Reference material: Papers assigned in the class.



Topics and Materials (Tentative, will be revised FREQUENTLY)

Additional References

Week 1

General introduction

· History and background introduction.

· Course introduction.

· Logistics


Quiz 1 (FAQ)



· Chapter 1 of GBC (the main textbook)


Paper presentation (sample, no review required)

· [Presenter: Peng Chu] Deep Learning of Graph Matching, Zanfir & Sminchisescu, CVPR 2018.

· Deep learning. LeCun, Bengio, & Hinton, Nature 2015.

· Deep neural networks for acoustic modeling in speech recognition. Hinton, et al., IEEE Signal Processing Magazine  2012.

· Relational inductive biases, deep learning, and graph networks. Bataglia et al. 2018.




Week 2


· Related basic knowledge in optimization and information theory.

· Deep learning in computer vision.



· Chapters 2-5 of GBC

· Related chapters in Computer Vision: Algorithms and Applications, Richard Szeliski, 2010.

· Convex optimization by Boyd & Vandenberghe, 2004.





Last day to drop the course (seriously)


Week 3



Deep Forward Networks

· Deep architecture

· Feed forward

· Training


· Chapters 6 of GBC

· Rumelhart, Hinton, & Williams. Learning representations by back-propagating errors. Nature 1986.

· Glorot, Bordes, & Bengio. Deep sparse rectifier neural networks. AISTAT 2011.




Week 4




Regularization for Deep Learning

· Various regularization techniques



· Chapters 7 of GBC

· Geometric deep learning: going beyond Euclidean data. Bronstein et al. IEEE SPM 2017.





Week 5

Optimization for Deep Learning

· Gradient descent and structure of cost functions

· Batch normalization



· Chapters 8 of GBC


Paper presentation (Review assignment)

· [Presenter: Heng Fan] Group Normalization. Wu & He. ECCV 2018

· [Presenter: Xinyi Li] On the Convergence of Adam and Beyond. Reddi, Kale, & Kumar. ICLR 2018.

· A fast learning algorithm for deep belief nets. Hinton, Osindero, & Teh. Neural Comp. 2006.

· Greedy layer-wise training of deep networks. Bengio, Lamblin, Popovici, & Larochelle. NIPS 2006.





Week 6

Convolutional Neural Networks

· Convolutional layers

· AlexNet



· Chapters 9 of GBC

· ImageNet Classification with Deep Convolutional Neural Networks, Krizhevsky & Hinton, NIPS 2012.


Paper presentation (Review assignment)

· [Presenter: Shikai Fang] Deep Convolutional Networks on Graph-Structured Data, Henaff, Bruna, & LeCun, 2015.

· [Presenter: Fengjiao Li] Semi-supervised Classification with Graph Convolutional Networks, Kipf & Welling. ICLR 2017

· LeCun, et al. Handwritten digit recognition with a back-propagation network. NIPS 1990.

· Gradient-based learning applied to document recognition. LeCun, Bottou, Bengio, & Haffner. Proc. IEEE 1998.

· Visualizing and Understanding Convolutional Networks, Zerler & Fergus, ECCV 2014.




Week 7

 Popular CNN Architectures

· VGGNet, GoogLeNet, ResNet



· Very Deep Convolutional Networks for Large-Scale Image Recognition. Simonyan & Zisserman, ICLR 2015.

· Going Deeper with Convolutions. Szegedy et al. CVPR 2015.

· Deep Residual Learning for Image Recognition. He, Zhang, Ren, & Sun, CVPR 2016.


Paper presentation (Review assignment)

· [Presenter: Anish Shah] SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size. Iandola et al. ICLR 2017.

· [Presenter: Zhongdong Liu] Aggregated Residual Transformations for Deep Neural Networks. Xie et al. CVPR 2017.

· [Presenter: Jumanah Alshehri] MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. Howard et al. 2017.

· Network In Network. Lin, Chen, & Yan. ICLR 2014.

· Densely Connected Convolutional Networks, Huang, Liu, van der Maaten, Weinberger, CVPR 2017.





Week 8

Popular Toolboxes

· Caffe, pyTorch, Tensorflow, MXNet



· Related papers

· TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems, Abadi et al., OSDI 2016.

· Caffe: Convolutional Architecture for Fast Feature Embedding, Jia et al., 2014

· PyTorch

· MxNet


Quiz 2





Last day to withdraw the course


Week 9


Sequence Modeling

· Recurrent Neural Networks (RNN)

· Long Short Term Memory (LSTM)



· Chapters 10 of GBC


Project proposal due

· Electronic version due before class (email to me before the class starts)

· Using ICLR 2018 template, download.

· Requirement:

-        Strictly following the ICLR template, including font and page sizes

-        Should contain at least the following sections: (1) Introduction, (2) Proposed research, (3) Evaluation plan, and (4) References

-        Minimum three pages, excluding the references.


Paper presentation (Review assignment)

· [Presenter: Saman Enayati] Sequence to sequence learning with neural networks. Sutskever, Vinyals, & Le. NIPS 2014.

· [Presenter: Nicholas Rork Torba] Neural Machine Translation by Jointly Learning to Align and Translate. Bahdanau, Cho, & Bengio. ICLR 2015.

· Recurrent Neural Networks, Wikipedia.

· Long Short-Term Memory. Hochreiter, Sepp; Schmidhuber, Jürgen. Neural Computation, 1997.





Week 10

Practical Methodology

· Practical issues in modeling, training and data augmentation.



· Chapters 11 of GBC


Paper presentation (Review assignment)

· [Presenter: Sheng Zhang] Attention is All you Need. Vaswani et al. NIPS 2017.

· [Presenter: Sidra Hanif] Dynamic Routing Between Capsules. Sabour, Frosst, & Hinton. NIPS 2017.

· Fully Convolutional Networks for Semantic Segmentation. J. Long, E. Shelhamer, T. Darrell, CVPR 2015.





Week 11


· Brief summarization of sampled applications

· Computer vision tasks (detection, tracking, segmentation)



· Chapters 12 of GBC


Paper presentation (Review assignment)

· [Presenter: Yanlong Qiu] Focal Loss for Dense Object Detection. Lin, Goyal, Girshick, He, & Dollár. ICCV 2017.

· [Presenter: Xuening Xu] Fully-Convolutional Siamese Networks for Object Tracking. Bertinetto et al.  ECCV Workshop 2016


· Deep Learning for generic object detection: a survey. Liu et al. ECCV 2018.

· Region-based convolutional networks for accurate object detection and segmentation, Girshick, Donahue, Darrell, Malik. CVPR 2014.

· YOLO9000: Better, Faster, Stronger. Redmon & Farhadi, CVPR, 2017.




Week 12

Unsupervised Deep Learning

· Autoencoders

· Generative Adversarial Networks



· Chapters 14 of GBC


Paper presentation (Review assignment)

· [Presenter: Patrick Hammer] Wasserstein Auto-Encoders, Tolstikhin, Bousquet, Gelly, & Schoelkopf. ICLR 2018.

· [Presenter: Bingyao Huang] Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. Ledig et al. CVPR 2017.

· Reducing the dimensionality of data with neural networks. Hinton & Salakhutdinov, Science 2006.

· Generative Adversarial Networks, Goodfellow et al., NIPS 2014.





Week 13

Advanced Topics (tentative)

· Deep reinforcement learning



· Chapters 12 of GBC


Paper presentation (Review assignment)

· [Presenter: Sandro Hauri] Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm. Silver et al. NIPS 2017.

· [Presenter: Terrel Nowlin] Neural Architecture Search with Reinforcement Learning. Zoph & Le. ICLR 2017





Week 14

Advanced Topics (tentative)

· Emerging topics


Project presentation






Project report due!