DeepMD training problem

### Summary

I am using DeepMD right now.
Now in licurve.out file errors are reduced. I am giving last few steps of lcurve.out

step rmse_val rmse_trn rmse_e_val rmse_e_trn rmse_f_val rmse_f_trn lr
If there is no available reference data, rmse_*_{val,trn} will print nan
......
1492000 6.66e-01 9.37e-01 7.04e-02 5.71e-02 6.04e-01 8.78e-01 1.1e-08
1493000 4.68e-01 7.73e-01 9.73e-02 5.93e-02 3.60e-01 6.81e-01 1.1e-08
1494000 8.44e-01 4.53e-01 3.80e-02 7.81e-02 7.82e-01 2.31e-01 1.1e-08
1495000 6.62e-01 6.34e-01 1.09e-01 1.13e-01 5.57e-01 5.20e-01 1.1e-08
1496000 9.56e-01 1.39e+00 1.16e-01 2.26e-01 8.53e-01 8.05e-01 1.1e-08
1497000 4.40e-01 8.34e-01 2.61e-02 4.64e-02 4.00e-01 7.63e-01 1.1e-08
1498000 1.23e+00 8.01e-01 2.10e-01 1.23e-01 1.03e+00 5.00e-01 1.1e-08
1499000 5.88e-01 4.31e-01 6.72e-02 7.78e-02 4.64e-01 1.92e-01 1.1e-08
1500000 1.06e+00 8.67e-01 7.30e-02 1.59e-01 9.88e-01 7.07e-01 1.0e-08

i am using 14 data set for each of which 80 percent is training and 20 percent for validation. datasets are increased also by bootstrapping. Now problem is that when I plot dft data and pred value plots are like

my code is also given below
{
"_comment1": " model parameters",
"model": {
"type_map": [
"Nb",
"Co"
],
"descriptor": {
"type": "se_e2_a",
"sel":"auto",
"rcut_smth": 0.5,
"rcut": 8.00,
"neuron": [
100,
200,
400
],
"resnet_dt": true,
"axis_neuron": 20,
"type_one_side": false,
"precision": "float64",
"seed": 1,
"_comment2": " that's all"
},
"fitting_net": {
"neuron": [
100,
100,
100
],
"resnet_dt": true,
"precision": "float64",
"seed": 1,
"_comment3": " that's all"
},
"_comment4": " that's all"
},

"learning_rate": {
"type": "exp",
"decay_steps": 10000,
"start_lr":1e-4,
"stop_lr": 1e-8,
"_comment5": "that's all"
},
"loss": {
"type": "ener",
"start_pref_e": 0.02,
"limit_pref_e": 1,
"start_pref_f": 1000,
"limit_pref_f": 1,
"start_pref_v": 0,
"limit_pref_v": 0,
"_comment6": " that's all"
},
"training": {
"training_data": {
"systems": [
"Data_8new/data_0",
"Data_8new/data_1",
"Data_8new/data_2",
"Data_8new/data_3",
"Data_9new/data_0",
"Data_9new/data_1",
"Data_9new/data_2",
"Data_9new/data_3",
"Data_10new/data_0",
"Data_10new/data_1",
"Data_10new/data_2",
"Data_10new/data_3",
"Data_11new/data_0",
"Data_11new/data_2",
"Data_11new/data_3",
"Data_11new/data_4",
"Data_12new/data_0",
"Data_12new/data_2",
"Data_12new/data_3",
"Data_12new/data_4",
"Data_13new/data_0",
"Data_13new/data_2",
"Data_13new/data_3",
"Data_13new/data_4",
"Data_14new/data_0",
"Data_14new/data_1",
"Data_14new/data_3",
"Data_14new/data_4",
"Data_15new/data_0",
"Data_15new/data_1",
"Data_15new/data_3",
"Data_15new/data_4",
"Data_16new/data_0",
"Data_16new/data_1",
"Data_16new/data_3",
"Data_16new/data_4",
"Data_17new/data_0",
"Data_17new/data_1",
"Data_17new/data_2",
"Data_17new/data_4",
"Data_18new/data_0",
"Data_18new/data_1",
"Data_18new/data_2",
"Data_18new/data_4",
"Data_19new/data_0",
"Data_19new/data_1",
"Data_19new/data_2",
"Data_19new/data_4",
"Data_20new/data_0",
"Data_20new/data_1",
"Data_20new/data_2",
"Data_20new/data_4",
"Data_21new/data_0",
"Data_21new/data_1",
"Data_21new/data_2",
"Data_21new/data_4"
],
"batch_size": "auto",
"_comment7": "that's all"
},
"validation_data": {
"systems": [
"Data_8new/data_4",
"Data_9new/data_4",
"Data_10new/data_4",
"Data_11new/data_1",
"Data_12new/data_1",
"Data_13new/data_1",
"Data_14new/data_2",
"Data_15new/data_2",
"Data_16new/data_2",
"Data_17new/data_3",
"Data_18new/data_3",
"Data_19new/data_3",
"Data_20new/data_3",
"Data_21new/data_3"
],
"batch_size": "auto",
"numb_btch": 1,
"_comment8": "that's all"
},
"numb_steps": 1500000,
"seed": 10,
"disp_file": "lcurve.out",
"disp_freq": 1000,
"save_freq": 10000,
"_comment9": "that's all"
},
"_comment10": "that's all"
}

can you pls help where I am going wrong??

### DeePMD-kit Version

3.1.1

### Backend and its version

TensorFlow 2.19.1

### Python Version, CUDA Version, GCC Version, LAMMPS Version, etc

_No response_

### Details

Training problem occur, error decreases but deviation between DFT and potential predicted value. details are mentioned above