Distributed Attention-Enabled Multi-Agent Reinforcement Learning Based Frequency Regulation of Power Systems

Abstract

This paper develops a new distributed attention-enabled multi-agent reinforcement learning method for frequency regulation of power systems. Specifically, the controller of each generator is modelled as an agent, and the reward and observation are designed based on the characteristics of power systems. All the agents learn their own control policies in the offline training phase and generate frequency control signals in the online execution phase. The target of the proposed algorithm is to conduct both offline training and online frequency control in a distributed way. To achieve this goal, two distributed information-sharing mechanisms are proposed based on the different global information to be discovered. First, a consensus-based reward-sharing mechanism is designed to estimate the globally averaged reward. Second, a distributed observation-sharing scheme is developed to discover the global observation information. Furthermore, the attention strategy is embedded in the observation-sharing scheme to help agents adaptively adjust the importance of observations from different neighbors. With these two mechanisms, a new distributed attention-enabled proximal policy optimization (DAPPO) based method is proposed to achieve model-free frequency control. Simulation results on the IEEE 39-bus system and the NPCC 140-bus system demonstrate that the proposed DAPPO achieves stable offline training and effective online frequency control.