Seeking Advice on Mitigating Thought Process Hallucinations in CHORD Training for RLVR

Hi,

I am using the CHORD training method from ms-swift for RLVR, specifically for fine-tuning a VLM model. I have encountered two issues and would appreciate your suggestions:

Thought Process Hallucinations:

While the training accuracy on my dataset is very high, the model’s reasoning process exhibits hallucinations and self-consistency issues. For instance, the final answer may be correct, but the reasoning process is clearly flawed or inconsistent.

Additional details:

I used the following parameters in ms-swift:

chord_sft_per_device_train_batch_size 1
chord_enable_phi_function false
chord_mu_warmup_steps 0
chord_mu_decay_steps 200
chord_mu_peak 0.9
chord_mu_valley 0.05

The model was trained for approximately 6,000 steps.

The SFT and RL datasets have little to no overlap, and the SFT dataset includes reference reasoning processes.

I would like to know if you have any recommendations to alleviate hallucinations in the reasoning process.

Dataset Overlap Between SFT and RL:

In the CHORD example code (link), the same dataset is used for both SFT and RL phases. However, since the training does not follow a complete SFT process (from policy shift to readaptation phase and then to overfitting phase), is it necessary to ensure no overlap between SFT and RL datasets, similar to the "SFT-then-RL" approach?

Thank you for your time and support!

Start with the CHORD example code referenced in the issue and review the reported training parameters, SFT/RL dataset relationship, and reasoning-quality symptoms. Done would require a project-grounded recommendation addressing thought-process hallucinations and whether dataset overlap is appropriate; no specific file or test is named.