Asynchronous Deep Reinforcement Learning: A Shared Experience Replay Framework

Abstract

Off-policy reinforcement learning (RL) algorithms with experience replay have achieved strong performance across a range of decision-making tasks. However, traditional implementations typically rely on a single agent interacting with one environment instance, which can limit exploration diversity and slow convergence. In this paper, we propose an asynchronous multi-agent RL framework that leverages a shared experience replay buffer. Each agent interacts independently with its own environment instance, contributing to a centralized buffer that aggregates diverse trajectories. This setup enhances sample diversity, accelerates learning, and scales efficiently with modern hardware. Our framework is compatible with standard off-policy algorithms such as Double DQN (DDQN) and DDPG, and we demonstrate its effectiveness across discrete and continuous control benchmarks. Experimental results show that our approach significantly improves convergence speed and learning stability compared to single-agent baselines. We discuss the theoretical implications of sharng experiences across agents and highlight real-world applications and future extensions, including hierarchical coordination, prioritized sampling, and deployment in real-time control systems.