| Month | Day | Before | Session | After | Deep.Dive |
|---|---|---|---|---|---|
| Sept. | 17 | ||||
| Sept. | 24 | Pre-Meeting Setup | Python: Application | ||
| Oct. | 1 | Python: Fundamentals | |||
| Oct. | 8 | Python: Functions and Classes | |||
| Oct. | 15 | Deep Learning Overview | To Deliver You From the Preliminary Terrors, The Intuitive Notion of the Chain Rule, What is a Neural Network? 0h18 , Gradient descent 0h20 , Analyzing our neural network | ||
| Oct. | 22 | What is backpropagation really doing? 0h12 , Backpropagation calculus 0h10, Backpropagation 2h25 | Discussion | ||
| Oct. | 29 | Backpropagation Cont. | Discussion | ||
| Nov. | 5 | Bigram Language Model 1h57 | Discussion | ||
| Nov. | 12 | Bigram Model Cont. | Discussion | Multilayer Perceptron Language Model 1h15 | |
| Nov. | 19 | No Session | Journal Club: BatchNorm Activations + BatchNorm 1h55 | ||
| Nov. | 26 | No Session (Thanksgiving week) | Backpropagation Deep Dive 1h55 | ||
| Dec. | 3 | But what is a GTP 0h27, Visualizing Attention 0h26 , Generatively Pretrained Transformer 1h56 | Discussion | How might LLMs store facts 0h22 | WaveNet 0h56 |
| Dec. | 10 | Transformers Cont. | Discussion | ||
| Dec. | 17 |
Fall Session: Preparation and Schedule
Schedule
Deep Learning
Edit: 9/26 Added to Schedule resources for
Derivatives (To Deliver You From the Preliminary Terrors (Thank you Kate Eastman!))
The Chain Rule (The Intuitive Notion of the Chain Rule)
Edit: 11/12 Updated schedule due to members conference/symposium and holiday travel.
Welcome!
Hello and welcome to our deep learning fall session! We are (re)iterating on the material we discussed over the summer so if this is the first time you’re joining us don’t worry about reading back through the older posts. If someone has directed you to this page and you’d like to be added to the mailing list, please get in touch with us.
What to Expect
We’ll be using python to go work from derivatives to generatively pretrained transformer models. Along the way we’ll talk about…
What deep learning models do
How deep learning models are trained
Architectural decisions to improve performance
Tools used for model development and refinement
Along the way we’ll also talk a fair bit about python, the logistical challenges that surround model training, and some of the ways models can learn the wrong things.
We’re interested in building understanding and intuition so as to avoid cases where you only think your model is working well. We’re going to encourage twin prescriptions of skepticism and caution in working with models. Instead of “move fast and break things” think “move slow and build things”.
What We Expect
We recommend viewing this group not as a class but a convening of autodidactic learners. Initially we’ll have a presentations coving foundational topics which will give way to open discussions. Having a list of thoughts and questions will be valuable to make the most of these sessions.
Since it’s really hard to train models on paper, we expect folks to be writing (and debugging) code. We recommend budgeting time for the videos and writing exploratory code on your own. It can be useful to write the code featured in the videos in parallel with them, which may substantially increase the amount of time the materi
al requires (and one’s comfort and fluency with these topics). Futher, applying or extending what you have seen to small data sets is a great way to solidify understanding.
Finally, we recommend identifying at least one peer that you can partner with. Explaining a new concept is a great way to solidify your understanding and having someone you can ask for a fresh perspective from is hugely useful. Although we’re interested in technical knowledge, we also want to increase connections between labs and the folks in them.
In summary:
Keep an eye on the upcoming materials
Write code, make mistakes, debug code
Make friends, ask for help, and teach others
What We Have Planned
Below is our “working draft” for the fall. Based on feedback from the summer session we’re beginning with python so doesn’t add friction when we use it to build models. We may also make a few additions to these topics (such as working with images, sequences, or modeling distrbutions) or add presentations on folks planned or ongoing projects to talk about how these model go from the classroom to the lab.
Preparing for the First Session:
To get started you’ll want to have installed python and a way to edit jupyter notebooks1.
For Python I recommend installing Anaconda if you’re on Windows and micromamba if you’re on Linux. For help installing Anaconda, see this link.
Although not strictly necessary I also recommend installing Visual Studio Code and Microsoft’s Live Share plugin. The former provides a nice editor for notebooks and the latter a way to collaboratively edit a code (think “google docs for code” or Teletype).
Footnotes
These behave similarly to R markdown / Quarto files for those coming from the R ecosystem.↩︎