Can't reproduce the same performance

Hello,
I trained base model and refinement model with all datasets(ytb_vos, vid, coco, det),
I think these datasets are LD(Large Dataset ).
So the final EAO should be 0.423 but I got only EAO =0.35.
Only different setting is batch size, original batch size is 64,
my batch size is:
- base_model training batch size : 64(e1 ~ e11), 16(e11 ~ e20)
- refinement training batch size : 32(e1 ~ e20)

Could the batch size make the large performance difference like this?
Could anyone give me any advice to reproduce the same performance?
I would appreciate in advance.

-----
I describes my results below.

**Reference**
VOT 2018 test results with the provided models ==>I got the same results
provieded models=SiamMask_VOT_LD.pth, SiamMask_VOT.pth

```
------------------------------------------------------------------------------------
| Tracker Name | Accuracy | Robustness | Lost Number | EAO |
------------------------------------------------------------------------------------
| Custom_mask_refine_SiamMask_VOT_LD | 0.598 | 0.229 | 49.0 | 0.422 |
| Custom_mask_refine_SiamMask_VOT | 0.609 | 0.276 | 59.0 | 0.380 |
------------------------------------------------------------------------------------
```
**My results**
1. training base for 20 epochs ==> epoch 8 was best EAO =0.314
In my case I get CUDA Out of Memory error at e11, So I reduced batch size 64->16 for e11~e20

```
---------------------------------------------------------------------------
| Tracker Name | Accuracy | Robustness | Lost Number | EAO |
----------------------------------------------------------------------------
| Custom_mask_checkpoint_e8 | 0.604 | 0.393 | 84.0 | 0.314 | ** best
| Custom_mask_checkpoint_e10 | 0.612 | 0.403 | 86.0 | 0.313 |
| Custom_mask_checkpoint_e6 | 0.602 | 0.417 | 89.0 | 0.298 |
| Custom_mask_checkpoint_e9 | 0.601 | 0.421 | 90.0 | 0.297 |
| Custom_mask_checkpoint_e7 | 0.601 | 0.440 | 94.0 | 0.292 |
| Custom_mask_checkpoint_e5 | 0.594 | 0.403 | 86.0 | 0.290 |
| Custom_mask_checkpoint_e11 | 0.601 | 0.435 | 93.0 | 0.287 |
| Custom_mask_checkpoint_e3 | 0.607 | 0.496 | 106.0 | 0.274 |
| Custom_mask_checkpoint_e4 | 0.603 | 0.478 | 102.0 | 0.271 |
...
```
2. tuning hyper parameters => penalty_k=0.1, windows_influence=0.48, lr=0.3 were best
```

---------------------------------------------------------------------------------------------------------------------------
| Tracker Name | Accuracy | Robustness | Lost Number | EAO |
---------------------------------------------------------------------------------------------------------------------------
| Custom_checkpoint_e8_r255_penalty_k_0_100_window_influence_0_480_lr_0_300 | 0.571 | 0.300 | 64.0 | 0.348 | **best
| Custom_checkpoint_e8_r255_penalty_k_0_080_window_influence_0_380_lr_0_300 | 0.580 | 0.328 | 70.0 | 0.342 |
| Custom_checkpoint_e8_r255_penalty_k_0_060_window_influence_0_380_lr_0_300 | 0.579 | 0.361 | 77.0 | 0.339 |
| Custom_checkpoint_e8_r255_penalty_k_0_120_window_influence_0_380_lr_0_250 | 0.580 | 0.295 | 63.0 | 0.338 |
| Custom_checkpoint_e8_r255_penalty_k_0_040_window_influence_0_460_lr_0_300 | 0.577 | 0.323 | 69.0 | 0.338 |
| Custom_checkpoint_e8_r255_penalty_k_0_080_window_influence_0_380_lr_0_400 | 0.582 | 0.351 | 75.0 | 0.335 |
| Custom_checkpoint_e8_r255_penalty_k_0_060_window_influence_0_420_lr_0_350 | 0.578 | 0.337 | 72.0 | 0.334 |
...
```
3. training base_model(epoch8) + refinement for 20 epochs ==> epoch10 was best with EAO=0.35
for all epochs, batch size = 32 is applied
for vot test, the hyper-parameters from Step2( set penalty_k=0.1, windows_influence=0.48, lr=0.3)
are applied on config_vot2018.json

```
-----------------------------------------------------------------------------------
| Tracker Name | Accuracy | Robustness | Lost Number | EAO |
-----------------------------------------------------------------------------------
| Custom_mask_refine_checkpoint_e10 | 0.571 | 0.290 | 62.0 | 0.350 |** final best results
| Custom_mask_refine_checkpoint_e9 | 0.568 | 0.290 | 62.0 | 0.347 |
| Custom_mask_refine_checkpoint_e6 | 0.578 | 0.300 | 64.0 | 0.344 |
| Custom_mask_refine_checkpoint_e4 | 0.593 | 0.309 | 66.0 | 0.343 |
| Custom_mask_refine_checkpoint_e12 | 0.562 | 0.295 | 63.0 | 0.341 |
| Custom_mask_refine_checkpoint_e5 | 0.572 | 0.300 | 64.0 | 0.340 |
| Custom_mask_refine_checkpoint_e15 | 0.573 | 0.318 | 68.0 | 0.339 |
| Custom_mask_refine_checkpoint_e19 | 0.581 | 0.314 | 67.0 | 0.338 |
| Custom_mask_refine_checkpoint_e13 | 0.559 | 0.304 | 65.0 | 0.337 |
...
```