DAMO-NLP-SG / DAMO-NLP-SG/LLM-R2

Lack of Evaluation Datasets

Open
#14 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
54
Forks
13
PR merge metrics
No merged PRs in 30d

Description

I encountered the same issue and found a related problem worth noting.

The error occurs because the evaluation dataset files (`sts-dev.csv`, `sts-test.csv`)
under `data/data_simcse/{dataset}/` are not included in the repository.
`prepare_CL_dataset.py` is responsible for generating these files, but it also requires
`training_triplets_{dataset}_total.csv` as input — which is likewise absent from the repo.

To reproduce the full training pipeline for datasets other than TPC-H (e.g., DSB, JOB),
could you provide:
1. The `training_triplets_*_total.csv` files (or the script used to generate them from the pool CSVs), and
2. The pre-trained SimCSE models for DSB and JOB, as used in the paper's experiments?

I would greatly appreciate your help in fully reproducing the results.
Thank you.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.