sensitivity variable being overwritten when fine-tuning 13B?

Nobody has claimed this yet.

https://github.com/lm-sys/FastChat/blob/51ed4fab89f61988e8395a3268595f1effb8528f/fastchat/train/train.py#L94

I used 8* gpu cards to fine tune 13B. It seems that there is a problem of mutual coverage of sentiments.

I have modified the code and added more log output, but I found that the presence is actually different, as if it has been overwritten by other graphics cards:

for j, sentence in enumerate(source):
role = roles[sentence["from"]]
print(conv.roles)
print(role)
print(conv.roles[j % 2])
print(role == conv.roles[j % 2])
print(sentence)
assert role == conv.roles[j % 2], f"{conv.roles}## {role}## {conv.roles[j % 2]} ##{sentence} ##{role} != {conv.roles[j % 2]}"
conv.append_message(role, sentence["value"])

my log ：
('USER', 'ASSISTANT')
ASSISTANT
ASSISTANT
True
{'from': 'gpt', 'value': 'Great! To create a leaderboard in HTML/CSS/JS, you can follow these steps:\n\n1. Fetch the data from the API using JavaScript. You can use the `fetch` API or any other
library to make the request. Once you have the data, you can parse the JSON using the `JSON.parse()` method.\n2. Sort the data by points in descending order. You can use the `sort()` method to s
ort the array of objects.\n```javascript\ndata.sort((a, b) => b.points - a.points);\n```\n3. Create a table in HTML to display the data. You can use the `` element and add the necessary t
able headers (``).\n```html\n`) and table rows (`\n \n \n \n \n \n \n \n \n \nRankNamePoints\n```\n4. Use JavaScript to loop through the sorted data and create a new table row for each object. You can use the `innerHTML` property to set the contents of the
`` element.\n```javascript\nconst leaderboard = document.getElementById("leaderboard");\ndata.forEach((player, index) => {\n const row = `\n \n ${index + 1}\n <
td>${player.name}\n ${player.points}\n \n `;\n leaderboard.innerHTML += row;\n});\n```\n5. Finally, you can add some CSS styles to make the leaderboard look more app
ealing. You can use CSS to add some background color, font, padding, and borders to the table.\n```css\ntable {\n border-collapse: collapse;\n width: 100%;\n}\n\nth, td {\n text-align: left;\
n padding: 8px;\n}\n\nth {\n background-color: #f2f2f2;\n font-size: 18px;\n font-weight: bold;\n}\n\ntr:nth-child(even) {\n background-color: #f2f2f2;\n}\n\ntd:first-child {\n font-weight
: bold;\n}\n\ntd:last-child {\n font-weight: bold;\n color: green;\n}\n```\nThat\'s it! With these steps, you should have a nicely formatted leaderboard displaying the data from the API.'}

AssertionError: ('USER', 'ASSISTANT')## USER## ASSISTANT ##{'from': 'human', 'value': 'i continue my dash through camp dragging the limp form of the dead orc in my jaws as i disappear into the
forest, i will run about 200 meters then drop the body and shapechange into an owl and fly up and sit on a branch looking down at the dead orc'} ##USER != ASSISTANT

Has anyone encountered a similar problem?

Start at fastchat/train/train.py around line 94 and reproduce the reported 13B fine-tuning run across eight GPUs. Compare the logged conversation roles and source sentences across workers to determine where the mismatch appears. Done means the role inconsistency is reproduced and its cause and expected behavior are established.

Written by the indexing model from the issue text.