Reddit

## Programming & Computing Sub-Reddits

Dataset URL - [awesome list of programming subreddits](https://github.com/iCHAIT/awesome-subreddits) [Code Pile Spreadsheet](https://docs.google.com/spreadsheets/d/1OrOnv-Cv1wRq0jNk4AegHiMtLk88YQDz5b1TP-o5SE8/edit#gid=1020850625)
[Another list of programming subreddits](https://github.com/learn-anything/reddit#linux) Thanks to @ncoop57!

Does the dataset exist in a scraped format ?

No, we need to format them into a dialogue format.

## Description
Obtain data from Pushift Reddit using wgets/http requests from 2009-2022 and filter for programming-related subreddits.

## Procedure
* [x] Obtain data from [Pushift Reddit](https://files.pushshift.io/reddit) from the years 2006-2022. We probably need to write a script that issues wgets for data dumps.
* [x] Store data dump on a GCP Bucket.
* [ ] Create 3 tables authors, submissions, and comments in BigQuery from the GCP Buckets.
* [ ] Merge posts with reply chains and author metadata (specifically bio)
* [ ] (Optionally) Filter for long dialogue chains following [OPT](https://arxiv.org/pdf/2205.01068.pdf)
* [x] Process Reddit threads (posts and replies) into a conversational form using [this script](https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit)
* [x] Filter for programming subreddits in the list of subreddits. Then we process non-programming subreddits and programming subreddits separately.
* [x] Process into output format ``{"text": string, "meta": obj}``
* [ ] Run dedup [Min-Hash](https://spark.apache.org/docs/2.2.3/ml-features.html#minhash-for-jaccard-distance)
* [x] Run ``lm_format`` [script](https://github.com/EleutherAI/lm_dataformat)

## Final Data Format inside ``text``
```
[Context]:
"Learning to learn", using deep learning to design the architecture of another deep network: https://arxiv.org/abs/1606.04474
[Response]:
using deep learning with SGD to design the learning algorithms of another deep network *

Extra Contexts:
[context/2]:
Could someone there post a summary of the insightful moments.
[context/1]:
Basically L2L is the new deep learning.
[context/0]:
What's "L2L" mean?

Other features:
[context_author]:
goodside
[response_author]:
NetOrBrain
[subreddit]:
MachineLearning
[thread_id]:
5h6yvl
```