Investigating Partner Diversification Methods in Cooperative Multi-agent Deep Reinforcement Learning

The 27th International Conference on Neural Information Processing. ICONIP 2020

Rujikorn Charakorn, Poramate Manoonpong, and Nat Dilokthanakul.

Access the paper here (pdf or pdf)


author="Charakorn, Rujikorn
and Manoonpong, Poramate
and Dilokthanakul, Nat",
editor="Yang, Haiqin
and Pasupa, Kitsuchart
and Leung, Andrew Chi-Sing
and Kwok, James T.
and Chan, Jonathan H.
and King, Irwin",
title="Investigating Partner Diversification Methods in Cooperative Multi-agent Deep Reinforcement Learning",
booktitle="Neural Information Processing",
publisher="Springer International Publishing",
abstract="Overfitting to learning partners is a known problem, in multi-agent reinforcement learning (MARL), due to the co-evolution of learning agents. Previous works explicitly add diversity to learning partners for mitigating this problem. However, since there are many approaches for introducing diversity, it is not clear which one should be used under what circumstances. In this work, we clarify the situation and reveal that widely used methods such as partner sampling and population-based training are unreliable at introducing diversity under fully cooperative multi-agent Markov decision process. We find that generating pre-trained partners is a simple yet effective procedure to achieve diversity. Finally, we highlight the impact of diversified learning partners on the generalization of learning agents using cross-play and ad-hoc team performance as evaluation metrics.",