CS 391 - Selected Topics: Game Artificial Intelligence Readings

---

Each reading assignment should be completed before the class on the date indicated.  These readings are subject to change; check here for updates.  If a reading assigned in class does not match the reading assignment here, the reading assigned in class supercedes.

• 1/24: R.S. Sutton, A. G. Barto. Reinforcement Learning: an introduction.  Chapter 2: intro-2.1.  Topics: evaluative (unsupervised) versus instructive (supervised) learning, n-armed bandit problem, exploration-exploitation balance.
• 1/26: R.S. Sutton, A. G. Barto. Reinforcement Learning: an introduction.  Sections 2.2-2.3.  Topics: actions, rewards, action-value function Q, epsilon-greedy action selection, softmax action selection.
• 1/31: Auer, N. Cesa-Bianchi, P. Fischer. P. Finite-time Analysis of the Multiarmed Bandit Problem.  (See also here.)  Sections 1 (skipping "In their classical paper ... mild assumptions." section), 2 (special attention to the algorithms), 4, and 5.  Topics: regret, upper confidence bounds (UCBs), UCB action selection algorithms.
• 2/2: R.S. Sutton, A. G. Barto. Reinforcement Learning: an introduction.  Chapter 3: intro-3.3, 3.6-3.8.  Topics: reinforcement learning problem, agent, environment, state, action, reward, returns, Markov decision processes (MDPs), state-value function V, action-value function Q, Bellman equation, optimal policy, optimal value-functions, Bellman optimality equation, backup diagrams.
• 2/7: R.S. Sutton, A. G. Barto. Reinforcement Learning: an introduction.  Chapter 6: intro-6.5.  Topics: Temporal-difference (TD) learning, TD prediction, bootstrapping, tabular TD(0), SARSA, Q-learning.
• 2/9: R.S. Sutton, A. G. Barto. Reinforcement Learning: an introduction.  Chapter 7: intro-7.6.  Topics: Eligibility traces, forward/backward view of TD(λ), SARSA(λ), Q(λ).
• 2/14: R.S. Sutton, A. G. Barto. Reinforcement Learning: an introduction.  Chapter 8: intro-8.4.  Topics: Generalization, function approximation, mean-squared error, on-policy distribution, gradient-descent, linear function approximation, binary features, coarse coding, tile coding, radial basis functions, Kanerva coding, linear gradient-descent SARSA/Watkin's Q learning.
• 2/16: G. Tesauro.  Practical Issues in Temporal Difference Learning.  Topics: Backgammon application of TD(λ)
• 2/21: G. Tesauro. Temporal Difference Learning in TD-Gammon.  Topics: Backgammon application of TD(λ)
• 2/23: L. Kocsis, C. Szepesvari. Bandit based Monte-Carlo Planning.  Topics: rollout-based planning, UCT (UCB applied to Trees).
• 2/28: S. Gelly, D. Silver. Combining Online and Offline Knowledge in UCT.  Topics: RAVE heuristic, linear value function approximation with TD(0)
• 3/1: R. Coulom.  Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. 
• 3/6: R. Lorentz, Amazons Discover Monte Carlo, G. Van Eyck, M. Müller.  Revisiting Move Groups in Monte Carlo Tree Search. 
• 3/8: H. Baier, M. Winands. Time Management for Monte-Carlo Tree Search in Go.
• 3/20: T. Cazanave. Nested Monte-Carlo Search. 
• 3/22: C. Rosin.  Nested Rollout Policy Adaptation for Monte Carlo Tree Search.
• 4/3: Essentials of Game Theory, Ch. 1, 2, Definition 3.2.1, Section 3.5; Handout: "Introduction to Counterfactual Regret Minimization", Section 1; [Optional: S. Hart, A. Mas-Colell. A Simple Adaptive Procedure Leading to Correlated Equilibrium. (G'burg library) Sections 2-3; H. Kuhn. Simplified Two-Person Poker.]
• 4/5: Handout: "Introduction to Counterfactual Regret Minimization", Section 2; T. Neller, S. Hnath.  Approximating Optimal Dudo Play with Fixed-Strategy Iteration Counterfactual Regret Minimization, through Section 2  [Optional: M. Zinkevich et al. Regret Minimization in Games with Incomplete Information.]
• 4/10: Handout: "Introduction to Counterfactual Regret Minimization", Section 3; T. Neller, S. Hnath.  Approximating Optimal Dudo Play with Fixed-Strategy Iteration Counterfactual Regret Minimization, remaining sections.
• 4/12: Handout "Introduction to Counterfactual Regret Minimization", Section 4
• 4/17: Handout "Introduction to Counterfactual Regret Minimization", Section 5
• (Readings relevant to independent final project work thereafter)