hf_summary format issue

There seem to be an issue in aggregating different ranks from summarize_from_feedback mainly about the counting issue.

How does the summarize_from_feedback work:

Basically openai only provide comparison rank of pairs, we could assume the first element is the better response than the second response (and filter out pairs with the same scores, since the current reward collator doesn't support tie rank ):

(A, B), (B, C), (B, D)

For a perfect rank of 3 ( A > B > C), there should be pairs of (A, B), (A, C), (B, C) and by counting the frequency of the first element (higher ranked), we could regain the original rank of ( A > B > C), since A=2, B=1, C=0.

Issues:

However there's some edge cases such as tie and imperfect comparison in the dataset mainly:

1. imperfect comparison rank (A, B), (B, C), counting the pair would resulted in ties in A and B. This could be solved by building a reverse linked list and add the count of previous ancestor, basically recover the (A, C) comparison.

2. tie results where given a pair of (A, B), (A, C), (B, D), (C, D) we could only infer A > B, C > D. Currently the format requires a list of response where each element must be a lower rank the previous one. So this needed to be address, if this happens quite frequently.

repeated summaries

As reported by @andreaskoepf , there seems to be a repeated summary in the final aggregated dataset.

floaty toys down a river.  I asked her if she wanted to buy one because the winner won 50 dollars.  She signed up one for her and her sister.  It was her idea to buy one for her sister.  I paid for it because she didn't have cash.  Now fast forward to 2 months later, the second time I remind her that she still owes me money, and I have her rubber ducks.  I held onto the ducks until I got my money.\n\nShe just verbally assaulted me.  She denies ever wanting to participate in the toy race and says I pushed it on her, so I shouldn't be paid.  She was quite rude and disrespectful.  I'm calling bullshit because I would never sign someone up if they hadn't fully agreed.  It was her idea to include her sister!  \n\nNow she owes me 6 dollars and I'm angry.  I would have let it go despite the rudeness and disrespect.  I am now going to do everything in my power to ensure she pays me even if I have to burn down the bridge between us.  In my mind it is already half burnt.\n\nWhat should I do to get my rude, disrespectful and cheapskate RA to pay me the money she owes?  I'm not afraid to be mean.\nTLDR;"]
[8] Continuations (2):
[8.0]: RA was rude, disrespectful and cheapskate.  Now owes me a lot of money.  I want to get her to pay me back.  What should I do?
[8.1]: RA verbally abused me and I want to get her to pay me.

[9] Prefix: ["A few months ago, I asked my RA if she wanted to participate in a fundraiser where we raced floaty toys down a river.  I asked her if she wanted to buy one because the winner won 50 dollars.  She signed up one for her and her sister.  It was her idea to buy one for her sister.  I paid for it because she didn't have cash.  Now fast forward to 2 months later, the second time I remind her that she still owes me money, and I have her rubber ducks.  I held onto the ducks until I got my money.\n\nShe just verbally assaulted me.  She denies ever wanting to participate in the toy race and says I pushed it on her, so I shouldn't be paid.  She was quite rude and disrespectful.  I'm calling bullshit because I would never sign someone up if they hadn't fully agreed.  It was her idea to include her sister!  \n\nNow she owes me 6 dollars and I'm angry.  I would have let it go despite the rudeness and disrespect.  I am now going to do everything in my power to ensure she pays me even if I have to burn down the bridge between us.  In my mind it is already half burnt.\n\nWhat should I do to get my rude, disrespectful and cheapskate RA to pay me the money she owes?  I'm not afraid to be mean.\nTLDR;"]
[9] Continuations (3):
[9.0]: RA was rude, disrespectful and cheapskate.  Now owes me a lot of money.  I want to get her to pay me back.  What should I do?
[9.1]: My rude RA voluntarily participated in a fundraiser and won't pay me back.  How do I get my money?
[9.2]: RA verbally abused me and I want to get her to pay me.

note the 8 and 9 shares the same summary

No files or tests are named. Start by tracing how the summarize_from_feedback dataset reaches the current reward collator, then inspect the aggregation logic for incomplete rankings, ties, and repeated summaries. Done means the supported format handles the reported ranking cases and the duplicate summary in the aggregated dataset is addressed.