huggingface / huggingface/datasets

Python Programming Puzzles

Open
#2,499 2 comments 2 reactions 0 assignees View on GitHub
dataset request
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

## Adding a Dataset
- **Name:** Python Programming Puzzles
- **Description:** Programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis
- **Paper:** https://arxiv.org/pdf/2106.05784.pdf
- **Data:** https://github.com/microsoft/PythonProgrammingPuzzles ([Scrolling through the data](https://github.com/microsoft/PythonProgrammingPuzzles/blob/main/problems/README.md))
- **Motivation:** Spans a large range of difficulty, problems, and domains. A useful resource for evaluation as we don't have a clear understanding of the abilities and skills of extremely large LMs.

Note: it's a growing dataset (contributions are welcome), so we'll need careful versioning for this dataset.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.