Princeton University Computer Science Department

Computer Science 598D Systems and Machine Learning

Spring 2022

 

Summary

In this graduate seminar, we plan to read and discuss recent research papers in the intersection between systems and machine learning (ML). We plan to focus on papers in two categories: systems for ML and ML for systems. The first category includes papers about building efficient hardware such as accelerators and software systems such as parallel, distributed, mobile, secure, privacy-preserving systems for ML. The second category includes papers on how to apply ML approaches to systems designs including new data structures, algorithms, and features.  We plan to invite internal (within the department) and external speakers to give guest lectures to give introductory and state-of-the-art developments in the area of systems and ML.

 

This course is open to graduate students. For seniors who are interested in taking this course, it requires permission from the instructors.  This course has no P/D/F option. All students are required to present papers and participate discussions, and complete programming assignments and a final project.   For final projects, students have the option to work in pairs.

 

This course satisfies the systems programming requirement.

Administrative Information

• Classes Times: TTh 9:30-10:50am
• Classroom: CS 105
• Zoom participation ID: https://princeton.zoom.us/j/98549560559?pwd=UVloNW1JZXpHSmx5UDBrOUwvcHdsUT09
• Instructor: Professor Kai Li (li@cs.princeton.edu)
• Teaching assistant: Yushan Su (yushans@princeton.edu)
  Office hours: appointments by email

Computing Resources

For some assignments and projects, laptops may be adequate.   If you need more compute power such as GPUs for your assignments and projects, we recommend two cloud resources:

·      Free Google Colab.  Please read its FAQ for how to use it and its limitations. 

·      Microsoft Azure.  Microsoft generously provides educational credits for our department. Which allow each student receive certain amount of Azure credits. If you need to use Microsoft Azure, you can send a request to our teaching assistant Yushan Su (yushans@princeton.edu). Please carefully manage the instance on Azure and stop the instance that is not running.  Here is the Azure tutorial: https://azure.microsoft.com/en-us/get-started/. Additional tips of using Azure will be provided in the courses.

 

Reading and Presentations

During each class meeting, we will have either a lecture by the instructor, guest speakers or student presentations and discussions of selected papers.  The tentative schedule below lists tentative topics for each class and suggested papers.  

 

Students taking the course are expected to read all selected papers.  Each student writes a brief review for each paper to be presented (one or two sentences to summarize the paper, one or two sentences to summarize the strengths, weaknesses and future directions of the paper). Please download the template here.  Each review should be submitted before its presentation and discussion.

 

To motivate students to read and discuss papers in depth, we ask a team of 3 students to present a paper and lead discussion together.  There are 3 parts for each presentation: 

1.     Problem statement, a brief overview of the state-of-the-art

2.     Key idea of the paper

3.     Strengths, weaknesses and future directions

The 3rd part includes class discussions and the student team serves as a “panel” for the discussions. 

 

The total time allocated for each paper is 40 minutes.  We suggest you prepare in advance and do not exceed 40 minutes to leave 40 minutes for another team.

 

We welcome students to suggest their interested papers.  If students are interested in substitute papers, after signing up for desired time slots, each team should discuss with the instructor at least one week in advance to have time to make announcement. 

 

Warmup

The goal of this warmup is to get familiar with ML frameworks and your computing resources (e.g. Microsoft Azure).  Since some students may have done programming in Pytorch and have experience with using Microsoft Azure, this assignment is not required and will not be graded.  We strongly encourage all students without experience complete the MNIST classification warmup exercise.  A related paper is

 

·      Gradient-Based Learning Applied to Document Recognition. Y. LeCun, L. Bottou, Y. Bengio and P. Haffner:, Proceedings of the IEEE, 1998. 

 

MNIST dataset will be automatic downloaded if using  MNIST classification. If you use another framework instead of Pytorch and need to download data separately, you can download the MNIST dataset (by Yann LeCun, Corinna Cortes and Christopher Burges). 

 

We have also set up the same on Google Colab:
https://colab.research.google.com/drive/1wUWfzIY81x7R6Xe8WZUkgzFoKmTws7co?usp=sharing

 

We encourage you to use the warmup to get familiar with using the cloud computing resources. 

 

You are welcome to use online help or getting help from our TA or other students. 

 

Programming Assignment 1 (Systems for ML)

There are two options for this programming assignment.  Each option is related to a published paper and based on its released open-source code.  You need to select one option and complete the assignment based on its requirements.  The two options are:

·      System for ML: Network pruning  (please click to see details)

·      System for ML: Binary ConvNet (please click to see details)

 

 

Programming Assignment 2 (ML for Systems)

Similar to programming assignment 1, there are two options for this assignment.  Each option is related to a published paper and based on its released open-source code.  You need to select one option and complete the assignment based on its requirements.  The two options are:

 

·      ML for System: Auto Neural Arch Search (please click to see details)

·      ML for System: Adaptive learned Bloom filter (please click to see details)

 

Final Project

For the final project, there are two options:

1.     Extend one of the four programming assignments above (work alone).  The extension can be either trying a modified algorithm or evaluating with different datasets.

2.     Work on a new project (work alone or in pairs).  You can extend the explorations or evaluations of a recently published paper. 

For option 2 with a two-student team, we expect your project is more significant such that each student can play different roles for the project.

 

Students are required to

·      Submit a brief project proposal (what you plan to do) in one slide.

·      Submit a final project report in the format of a conference paper, to concisely describe what you have done and your results.  For a two-student team (option 2), the final report should state clearly who did what.

·      Give a 10-min presentation for option 1 or a 15-min presentation for option 2. 

The final reports are due at 11:59pm of the Dean's day, which is the deadline of the university.  The final presentations will be scheduled soon after the Dean’s day.

 

Grading and Requirements

Students taking this graduate seminar for credits will be graded as follows:

• Presentation, reading and participation (30%)
• Assignments (30%)
• Final projects (40%)

Auditors are required to attend all lectures and participate in presentations, readings, and discussions.

 

 

 

Schedule (Tentative)

Dates	Presenters	Topics	Suggested papers	Submissions
1/25	Prof. Kai Li	Introduction (slides, recording)	A New Golden Age in Computer Architecture: Empowering the Machine- Learning Revolution. Jeff Dean, David Patterson, and Cliff Young, IEEE Micro, 38(2), 21-29.
1/27	Prof. Jia Deng, Princeton CS	Introduction to deep learning (slides, recording)	Deep Learning. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton, Nature 521, May 2015.
2/1	Yushan Su	Introduction to deep learning framework (slides, recording)	PyTorch: An Imperative Style, High-Performance Deep Learning Library. Adam Paszke, et al. TensorFlow: A System for Large-Scale Machine Learning. Martín Abadi, et al. OSDI 2016	Start Warmup
2/3	Prof. Jaime Fernandez Fisac, Princeton ECE	Introduction to Reinforcement Learning (recording)	Human-level control through deep reinforcement learning. Mnih, V. et al, Nature, 2015.  (Earlier version).   Reinforcement Learning: An Introduction. (book) Richard S. Sutton and Andrew G. Barto.  MIT Press 2018.
2/8	Prof. Kai Li	Intro to Systems for ML (recording)	Large Scale Distributed Deep Networks. Jeffrey Dean, et al. NIPS 2012. TensorFlow: A System for Large-Scale Machine Learning. Martín Abadi, et al. OSDI 2016 XLA: Optimizing Compiler for Machine Learning.	Start assignment 1 (Systems for ML)
2/10	Tianle Cai, Tong Wu   Huilan Li, Runze Liu, Qingchen Dang	Network pruning (recording)	The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, Jonathan Frankle, Michael Carbin. ICLR 2019.  The Lottery Tickets Hypothesis for Supervised and Self-Supervised Pre-Training in Computer Vision Models. Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang.  CVPR 2021.	Submit review
2/15	Hongjie Wang, Yu-An Shih     Dora Zhao, Haiyue Ma	Pruning and Binarizing (recording)	Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, Daniela Rus.  MLSys 2021. XNOR-Net++: Improved Binary Neural Networks. Adrian Bulat, Georgios Tzimiropoulos.  BMVC, 2019.	Submit review
2/17	Nick Kaashoek, John Yang       Alex Raistrick, Yiming Zuo	Learned data structures (recording)	[1] The Case for Learned Index Structures. T. Kraska, A. Beutel, E. H. Chi, J. Dean, and N. Polyzotis. SIGMOD 2018.    [2] A Model for Learned Bloom Filters and Optimizing by Sandwiching.  Michael Mitzenmacher, et al. NIPS 2018.   [3] ALEX: An Updatable Adaptive Learned Index Jialin Ding et al. SIGMOD 2020.	Submit review for [1] and [3]
2/22	Bri Butler, Tianle Cai   Huihan Li, Runze Liu	AutoML (recording)	AutoML-Zero: Evolving Machine Learning Algorithms From Scratch Esteban Real, Chen Liang, David R. So, Quoc V. Le, ICML 2020   Once For All: Train One Network and Specialize It for Efficient Deployment H. Cai, C. Gan, T. Wang, Z. Zhang, S. Han.  ICLR 2020.	Submit assignment 1 Start assignment 2 Submit review
2/24	John Yang, Qingchen Dang     Yu-An Shih, Kaifeng Lyu	Federated learning (recording)	Federated learning: Collaborative machine learning without centralized training data. H. Brendan McMahan and Daniel Ramage., 2017.   Inverting Gradients - How easy is it to break privacy in federated learning? (Privacy) Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, Michael Moeller, NeurIPS 2020	Submit review
3/1	John Thoppe (UCLA) and Prof. Ravi Netravali, Princeton	Distributed Training (recording)	Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads.  John Thorpe, Yifan Qiao, Jonathan Eyolfson, Shen Teng, Guanzhou Hu, Madison; Zhihao Jia, Jinliang Wei, Keval Vora, Ravi Netravali, Miryung Kim and Guoqing Harry Xu.  OSDI 2020.	Submit assignment 2 Check suggested final project
3/3	Tianle Cai, Kaifeng Lyu   Beiqi Zou, Tong Wu	Graph Neural Networks (recording)	Semi-supervised classification with graph convolutional networks.  Kipf, Thomas N., and Max Welling. 2016   A unified lottery ticket hypothesis for graph neural networks. Chen, Tianlong, et al. ICML 2021.   Binary Graph Neural Networks.  Bahri, Mehdi, Gaétan Bahl, and Stefanos Zafeiriou. CVPR 2021.	Submit review
3/4				Submit final project proposal (finish discussing with the instructor)
3/15	Prof. Danqi Chen, Princeton	Pre-trained language models (recording)	BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding  Devlin et al., 2018. RoBERTa: A Robustly Optimized BERT Pretraining Approach.  Liu et al., 2019. Language Models are Few-Shot Learners, Brown et al. 2020 Making Pre-trained Language Models Better Few-shot Learners, Tianyu Gao, Adam Fisch, Danqi Chen. 2021
3/17	Brian Max Tchouambe, Hongjie Wang   Ryan Torok, Yiming Zuo	Mixture of Experts (recording)	Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Fedus, William, Barret Zoph, and Noam Shazeer. 2021.   FastMoE: A Fast Mixture-of-Expert Training System. Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, Jie Tang. 2021.	Submit review
3/22	Nick Kaashoek, Beiqi Zou     Huihan Li, Runze Liu	Deduplication and K-NN Retrieval for Training (recording)	Deduplicating training data makes language models better.   K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini. 2021. Improving language models by retrieving from trillions of tokens. Sebastian Borgeaud, et al. 2022.	Submit review
3/24	Prof. Song Han, MIT	Tiny Deep learning (recording)	MCUNet: Tiny Deep Learning on IoT Devices J. Lin, W. Chen, Y. Lin, J. Cohn, C. Gan, S. Han.  NeurIPS’20, spotlight    MCUNet-v2: Memory-Efficient Inference for Tiny Deep Learning J. Lin, W. Chen, H. Cai, C. Gan, S. Han. NeurIPS’21   Network Augmentation for Tiny Deep Learning H. Cai, C. Gan, J. Lin, S. Han.  ICLR’22
3/29	Dr. Safeen Huda, Google Brain	Google TPU (recording)	In-Datacenter Performance Analysis of a Tensor Processing Unit.  N. Jouppi et al. ISCA 2017   The Design Process for Google's Training Chips: TPUv2 and TPUv3.  T. Morrie, et al. IEEE MICRO 2021.   Ten Lessons From Three Generations Shaped Google’s TPUv4i. Norman P. Jouppi, et al. ISCA 2021.   A Full-Stack Search Technique for Domain Optimized Deep Learning Accelerators.  Dan Zhang, Safeen Huda, Ebrahim Songhori, Kartik Prabhu, Quoc Le, Anna Goldie, Azalia Mirhoseini.  ASPLOS 2022.
3/31	Prof. Naveen Verma, Princeton	In-Memory Accelerator (recording)	Scalable and Programmable Neural Network Inference Accelerator Based on In-Memory Computing.   H. Jia, M. Ozatay, Y. Tang, H. Valavi, R. Pathak, J. Lee, and N. Verma.  IEEE J. of Solid-State Circuits (JSSC), 2021. In Memory Computing.   Naveen Verma, Hongyang Jia, Hossein Valavi, Yinqi Tang, Murat Ozatay, Lung-Yen Chen, Bonan Zhang, and Peter Deaville.  SSCS Magazin, 2019. In-Memory Computation of a Machine-Learning Classifier in a Standard 6T SRAM Array.  Jintao Zhang, Zhuo Wang, and Naveen Verma.  IEEE Journal of Solid-State Circuit.  2017
4/5	Nick Kaashoek, Brian max Tchouambe   John Yang,  Dora Zhao	ML and Database (recording)	Neural Databases. James Thorne, Majid Yazdani, Marzieh Saeidi, Fabrizio Silvestri, Sebastian Riedel, Alon Halevy.  PVLDB 2020.   Database reasoning over text. James Thorne, Majid Yazdani, Marzieh Saeidi, Fabrizio Silvestri, Sebastian Riedel, Alon Halevy. ACL 2021.	Submit review
4/7	Yu-An Shih, Haiyue Ma   Hongjie Wang, Qingchen Dang	Improving Software Stack (recording)	SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems.  Beidi Chen, Tharun Medini, James Farwell, Sameh Gobriel, Charlie Tai, Anshumali Shrivastava.  MLSys 2020.   Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization, Quantizations, Memory Optimizations, and More Shabnam Daghaghi, Nicholas Meisburger, Mengnan Zhao, Anshumali Shrivastava.  MLSys 2021.	Submit review
4/12	Dr. Yanqi Zhou, Google Brain (to be confirmed)	ML Compilers (slides)	A Learnt Performance Model for Tensor Processing Units (MLSys 2021) Transferable Graph Optimizers for ML Compilers (NeurIPS 2020) A Transferable Approach for Partitioning Machine Learning Models on Multi-Chip-Modules (MLSys 2022) Device Placement Optimization with Reinforcement Learning (ICML 2017) A Flexible Approach to Autotuning Multi-Pass Machine Learning Compilers (PACT 2021)
4/14	Bri Butler, Brian Max Tchouambe   Kaifeng Lyu Alexander Raistrick,	Self-Supervised Learning	Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin. NeurIPS 2020.   Self-supervised Pretraining of Visual Features in the Wild. Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, Piotr Bojanowski. 2021.	Submit review
4/19	Haiyue Ma,  Ryan Torok     Beiqi Zou, Tong Wu,	Learned prefetching and Leakage in NLP models	A Hierarchical Neural Model of Data Prefetching. Zhan Shi Akanksha Jain Kevin Swersky Milad Hashemi Parthasarathy Ranganathan Calvin Lin. ASPLOS 2021. Extracting Training Data from Large Language Models. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, Colin Raffel.  USENIX Security 2021.	Submit review
4/21	Alex Raistrick, Yiming Zuo Bri Butler, Dora Zhao	Self-Supervised Learning (slides)	Representation Learning via Invariant Causal Mechanisms. Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, Charles Blundell. ICLR 2021.   Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet? Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, Jovana Mitrovic. 2022.	Submit review Signup Final Presentation
5/3 (dean's day)				Submit final project report
5/4 (9:30am-12pm) and  5/5 (9:30am-12pm)	All students	Final project presentations	Final Presentation