[Question]: Difficulty Reproducing Results in CoT.ipynb

Describe the issue

I attempted to reproduce the results of the LLMLingua paper using the CoT.ipynb notebook from the examples folder. However, I encountered a discrepancy in the accuracy achieved. The result in CoT.ipynb reports 78% accuracy, but I only achieved 68% accuracy in my reproduction attempt.

Changes made to CoT.ipynb:

• Updated the openai library to 1.14.2
• Changed the "completions" API endpoint to "chat.completions".

The rest of the file was kept unchanged as per the GitHub version.

Expected Behavior:

The reproduction should yield results consistent with the reported 78% accuracy, as in the output of the last cell in the notebook:

num_q 1319 correct 1032 ratio 0.7824

Actual Behavior:

I obtained only 68% accuracy:

num_q 1319 correct 900 ratio 0.6823

Question

Is this expected? Any ideas what could be the problem here? If the culprit is the openai model used, any ideas how to fix this -- gpt-3.5 model family no longer allow non-chat inference...

Thanks in advance!