关键词: 过程控制, 深度强化学习, 双延时深度确定性策略梯度, 三角衰减周期

Abstract:

Deep reinforcement learning (DRL) algorithms have recently attracted considerable attentions in the field of industrial process control due to their strong ability to achieve optimal control policies through environment-agent interactions without relying on historical data or prior knowledge. Among a variety of DRL models, the twin delayed deep deterministic policy gradient (TD3) model can effectively address the problem of "Q-values overestimation" endured by the deep deterministic policy gradient (DDPG) model, establishing itself as a leading DRL model for industrial process control. However, the original TD3-based controller shows limitations in the industrial process control with considerable policy fluctuations, especially, the Q-values underestimation may result in suboptimal control policies. Accordingly, this study introduced an enhanced TD3 (ETD3) model to improve the performance of TD3 in practical industrial process control. In the ETD3 model, an evaluation criterion was firstly presented to assess the overestimation or underestimation of actor network parameters, and then the loss function that was input to the critic network was adjusted according to the assessment results. Subsequently, the fixed learning rate in the original TD3 model was replaced by a triangular decay cycle learning rate, which can enhance the model's training convergence and control performance. Finally, the effectiveness of the ETD3 model was verified by the performance of the ETD3 controller in the natural gas dehydration process under different disturbances.

Key words: process control, deep reinforcement learning, twin delayed deep deterministic policy gradient (TD3) model, triangular decay cycle