Week 10, 23–27 November
Welcome to week 10! This week we look at Bayesian logistic regression and the Laplace approximation.
As usual your mark this week comes from: completing the discussion task (10%), attempting the in-note questions (20%), and this week’s assessed questions (70%). Full details on the assessments page including rules you must know.
Office hours:
You can meet us on MS Teams in the Meet-up channel of the MLPR 2020/21 Chat team on Friday at 9:30 AM and 4:30 PM UK time (GMT). One of Arno or Iain will be there. If you want to discuss something individually, please contact us by email: Arno aonken@inf.ed.ac.uk or Iain i.murray@ed.ac.uk.
Here is what you need to do in Week 10:
Any catch-up: If there are any threads on hypothesis that didn’t get resolved (allow 48 hrs), email Arno and Iain.
Lecture notes: Work through the Week 10 notes, answering all the questions. It’s fine to make mistakes here, but an honest attempt at these by Friday at
4pm6pm (UK time) is required.Question sheet: Do the week 10 question sheet. This question sheet is assessed and forms the bulk of this week’s marks.
Tutorial group discussions: You should post at least one thing that you’d find it useful to go over with your group and/or tutor. Or, if you’re on top of everything, state in advance that you would be happy to answer questions from the group.
This week you’ll consider building a rating system that models how good players of a game are from outcomes of a two-player game. This model might be used to produce a “leaderboard” that ranks players, or it could be used to match players together on an online server. Microsoft’s TrueSkill is such a system (which also applies to multi-player games), another model is used by the KGS Go Server rankings.
We can build a model for \(D\) players using logistic regression. If in game \(n\) player 1 is \(a\) and player 2 is \(b\), we construct a vector \(\bx\nth = \bx^{(n,a)} - \bx^{(n,b)}\), where \(\bx^{(n,a)}\) and \(\bx^{(n,b)}\) are one-hot encodings of the player identities. So \(\bx\) contains a \(+1\) and \(-1\) indicating which two players played, and the remaining entries are zero. Standard logistic regression can then be used to model \(P(\text{player a wins}\g \bw, \bx) = \sigma(\bw^\top\bx)\).
Some things you could consider:
- How could we create a leaderboard from this model?
- In some games there is an advantage for “player 1”, such as the white player in chess, the home side in football. How could we allow our model to learn that?
- Given two players, we might be uncertain who will win because a) they are close in ability, b) we don’t know enough about them yet (we don’t have enough training data). Which of these would Bayesian logistic regression help with, and how can these two forms of uncertainty be represented and estimated?
- How would you match players together using such a model in practice? Think particularly about the cold start problem: how do we rapidly incorporate new users of a system (who might be experts from playing the game elsewhere), without annoying too many people?
- What might other practical issues with a model be? Are there any potential harms of such a system that we should be aware of and try to avoid?
- How would we model games that aren’t just win/lose like the game of Go? For example, chess has draws, and football has numbers of goals scored on each side.
- Sometimes a player can have a bad day (they fall ill, get disqualified, or have an accident), and might lose to someone much worse than them. How could we let our model (nearly) ignore such surprising results?
- What happens to our predictions if we add a constant (like 53) to every weight? If we want to “anchor” our rating system, so that a weight of 1 means roughly the same thing each time we refit our model, and over time, how might we do that?
- What are the pros and cons of replacing logistic regression with a neural network? How would we rank a leaderboard of players now? Might we be able to do anything with a neural network model that we can’t do with logistic regression?
Create a short summary of your conclusions for your tutor, probably as bullet points. This summary is (lightly) assessed! See the group instructions for details on how to submit the group discussion report.
We recommend that you aim to finish the questions (in the notes and question sheet) and submit your discussion report by the end of Thursday. We will assess only what you have submitted by 4pm 6pm UK time on Friday.
As in the Informatics late policy, extensions are not available for weekly hand-ins. We expect many students to miss or under-perform on one hand-in, and will discount the one with the lowest mark. If you experience more significant disruption to your studies, you may need to file special circumstances. Consult your Personal Tutor or Student Support team. Lecturers on a course cannot make allowances outside these procedures.