DynaMIX: Sample-Efficient Multi Agent Reinforcement Learning with Multi-step Temporal Forward Dynamics Modeling

In multi-agent reinforcement Learning (MARL), improving sample efficiency poses a significant challenge. Existing methods such as QMIX primarily focus on modeling agent cooperation but still face limitations, including the requirement for extensive interaction data and long training times. To address these challenges, existing studies have investigated the use of auxiliary tasks based on representation learning, including self-supervised learning approaches. In this paper, we propose DynaMIX, which incorporates a multi-step temporal forward dynamics modeling (MTFDM) as an auxiliary task for QMIX. DynaMIX forecasts the future state beyond the immediate next state. This enables explicit learning of environmental dynamics, allowing for better understanding and adaptation to complex multi-agent interactions. Experimental results on the StarCraft II micromanagement benchmark demonstrate that DynaMIX significantly improves sample efficiency under limited interactions compared to QMIX.