Performance of Reinforcement Learning in Optimizing the Sphere Function

Recently, I have been exploring the enhanced open-source optimization library NEORL by running several scripts from the examples folder, including ex_a2c.py, ex_mixedrl.py, ex_ppo.py, ex_ppoes.py, and ex_dqn.py.

The results obtained from running ex_de.py and ex_pso.py are as follows:

------------------------ DE Summary -------------------------- Best fitness (y) found: 4.5832503222971595e-07
Best individual (x) found: [-0.00046072881266614844, 3.602063204023731e-05, -0.00026397842517867395, -0.0004158335753320424, 4.641482675974058e-05]

------------------------ PSO Summary -------------------------- Best fitness (y) found: 2.479339135984705e-08
Best individual (x) found: [-8.039549316355327e-05, 6.975337826376289e-05, -8.102620453636055e-05, 1.1754584251176704e-05, 8.222533786578424e-05]

These results align with expectations.

However, when using reinforcement learning methods for optimization, the outcomes were different. The results from ex_dqn.py are:

--------------- DQN Results --------------- The best value of x found: [-1 -1 -1 -1 -1]
The best value of y found: 5

The results from ex_ppo.py are:

--------------- PPO Results --------------- The best value of x found: [ 1.19758829 -0.73083058 1.22654803 0.27194634 1.28856361]
The best value of y found: 5.2071023

The results from ex_ppoes.py are:

************************* PPO-ES Summary ************************* Best fitness (y) found: [-0.28533976514716974, 0.4912544114233979, -0.7090348289656816, 0.06294215001853487, 0.17958195999566628]
Best individual (x) found: 0.8616914616084251

Why do the results from reinforcement learning methods show such a deviation from the optimal solution? Your feedback would be greatly appreciated.