Lecture 1 - Overview of Stochastic Approximation
Lecture 2 - Estimating the Mean of a Random Variable
Lecture 3 - Estimating change rates of webpages
Lecture 4 - An Introduction to Reinforcement Learning
Lecture 5 - Policy Evaluation in Reinforcement Learning
Lecture 6 - Probability Spaces - A Measure Theoretic Perspective
Lecture 7 - Random Variables as Measurable Maps
Lecture 8 - Expectation of Random Variable as Lebesgue Integration
Lecture 9 - Conditional Expectation: A Formal Introduction
Lecture 10 - Properties of Conditional Expectation
Lecture 11 - Martingales: Definition and Examples
Lecture 12 - Doob’s Uncrossing Lemma
Lecture 13 - Doob’s Forward Convergence Theorem for Supermartingales
Lecture 14 - $L^2$ Martingales
Lecture 15 - Revisiting the Martingale Convergence Theorem via Stopped Processes
Lecture 16 - Existence and Uniqueness of Solutions to Ordinary Differential Equations
Lecture 17 - Continuity of ODE Solutions with Respect to Initial Conditions
Lecture 18 - Asymptotic Behaviour of Solutions to ODEs
Lecture 19 - Stability of ODEs and Lyapunov Methods
Lecture 20 - Foundations of Stochastic Approximation - Assumptions and Key Definitions
Lecture 21 - Almost Sure Convergence of Stochastic Approximation Iterates
Lecture 22 - Concluding the Convergence Proof: Internal Chain Transitivity, Connectedness, and Invariance
Lecture 23 - How Stochastic Approximation Iterates Track an ODE: The Key Lemma
Lecture 24 - Proof of the Key Lemma - Part I
Lecture 25 - Proof of the Key Lemma - Part II
Lecture 26 - Extensions, Variants, and Applications of Stochastic Approximation
Lecture 27 - Convergence Rate for Linear Stochastic Approximation - Part 1
Lecture 28 - Convergence Rate for Linear Stochastic Approximation - Part 2
Lecture 29 - Lower Bounds and the Minimax Risk in Estimation
Lecture 30 - Stability Requirements in Stochastic Approximation
Lecture 31 - Almost Sure Boundedness of Iterates: Theorem and Example
Lecture 32 - Distributed Estimation of the Mean of a Random Vector
Lecture 33 - Robust Distributed Learning under Adversaries
Lecture 34 - Motivation for Stochastic Recursive Inclusion in Distributed Mean Estimation
Lecture 35 - Well-Posedness of Differential Inclusions
Lecture 36 - Convergence of Distributed Robust Mean Estimation Iterates through the Lens of Differential Inclusions
Lecture 37 - Almost Sure Convergence via Robbins-Siegmund Theorem - Part 1
Lecture 38 - Almost Sure Convergence via Robbins-Siegmund Theorem - Part 2
Lecture 39 - Introduction to Reinforcement Learning and Value Function Approximation
Lecture 40 - Temporal Difference Algorithm Through the Lens of Stochastic Approximation
Lecture 41 - How Good Is the TD Solution? Fixed Point Analysis in Linear Approximation
Lecture 42 - Almost Sure Convergence Analysis of Temporal Difference Learning via the ODE Method
Lecture 43 - Best Policy Algorithm for Q-Value Functions: A Stochastic Approximation Formulation
Lecture 44 - Asymptotic Analysis of Q-Learning Algorithm
Lecture 45 - Asymptotic Behaviour of the Q-Learning Limit ODE - A Switching Systems Perspective
Lecture 46 - Concluding the Asymptotic Analysis of Q-Learning
Lecture 47 - Q-Learning with Linear Function Approximation - A Unified Switching Systems Perspective
Lecture 48 - Q-Learning with Linear Function Approximation under epsilon - Greedy Exploration
Lecture 49 - Analysis of Limiting Dynamics in Q-Learning with Function Approximation
Lecture 50 - Review of Probability Theory and Fundamental Inequalities
Lecture 51 - Review of Stochastic Approximation Concepts
