MARL Chapter 9.6.2 & 9.7: Policy Representations and Homogenous Agents

This meeting will continue the material from Chapter 9 in Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. Last time we covered how agent models can be used to calculate best response actions with a joint action value function. In Section 9.6.2, we will use those agent models in an alternative way to generate a compact representation of other agent policies. That representation can then be used to condition policy and value functions of our agents. Learning a policy representation can make algorithms more powerful that we have already addressed such as independent DQN and actor-critic policy training.

In section 9.7, we will explore another tool we can apply to previous algorithms in which we treat all agents as interchangeable. This simplification only works in environments where the information passed to our agents is relative to that agents perspective. That way all policy and value functions can share parameters as long as the game itself is solvable by agents that are interchangeable (they can still behave differently due to their unique position in the environment).

As usual you can find below links to the textbook, previous chapter notes, slides, and recordings of some of the previous meetings.

Meetup Links:
Recordings of Previous RL Meetings
Recordings of Previous MARL Meetings
Short RL Tutorials
My exercise solutions and chapter notes for Sutton-Barto
My MARL repository
Kickoff Slides which contain other links
MARL Kickoff Slides

MARL Links:
Multi-Agent Reinforcement Learning: Foundations and Modern Approaches
MARL Summer Course Videos
MARL Slides

Sutton and Barto Links:
Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto
Video lectures from a similar course