TD3 Tracking Performance Issue in a Nonlinear Control System

Hello,

I am working on a research project involving TD3-based trajectory tracking of a nonlinear Twin Rotor MIMO System (TRMS) in MATLAB.

I found the “Machine Learning for Motor Control” project and its suggested TD3-based PMSM control workflow. Although my system is different (TRMS rather than PMSM), I am using TD3 for continuous control and I am facing a tracking-performance issue.

For a step reference of 0.8 rad (pitch) and 0.9 rad (yaw), my TD3 agent eventually learns partial tracking, but the performance is inconsistent.

My latest results are:

Ep 400 | Reward = -5852.5 | StepErr = 0.7652
Ep 450 | Reward = -1785.0 | StepErr = 0.0472
Ep 500 | Reward = -1743.4 | StepErr = 0.1594
Ep 550 | Reward = -3019.5 | StepErr = 0.1932
Ep 600 | Reward = -1553.3 | StepErr = 0.0674

Final evaluation:

• Step RMSE: Pitch = 0.1200 rad, Yaw = 0.0784 rad
• Low-frequency sine RMSE: Pitch = 0.1502 rad, Yaw = 0.1045 rad
• High-frequency sine RMSE: Pitch = 0.1494 rad, Yaw = 0.1170 rad

Current TD3 settings:

• Gamma = 0.99
• Actor learning rate = 1e-3
• Critic learning rate = 1e-3
• Batch size = 256
• Tau = 0.005
• Policy noise = 0.2
• Noise clip = 0.5
• Action limit = ±2.5 V
• Observation dimension = 15

My reward function is:

R = -10(e_pitch² + e_yaw²) - 0.001(u1² + u2²)

The main question is: what would you recommend checking first in a TD3 implementation when the agent can sometimes achieve good tracking during training but does not maintain it consistently?

In particular, I would appreciate guidance on whether I should investigate the reward function, learning rates, exploration noise, observation design, or the TD3 implementation itself.

I can provide the relevant MATLAB code and additional results if needed.

Thank you.

No repository file, test, or entry point is named, and no MATLAB code is included. Start by reviewing the TD3 implementation, reward function, exploration noise, and observation design against the reported training and evaluation metrics. Done would require identifying and documenting a reproducible cause of the inconsistent tracking, with consistently improved RMSE results.