google-deepmind / google-deepmind/streamingqa

Tool for splitting dataset by `publish_date` range?

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
51
Forks
0
PR merge metrics
No merged PRs in 30d

Description

It would be nice to have a tool for splitting up the dataset into any arbitrary `t=t_1 to t=t_2`.

For instance, if we index the corpus along the publish date, then created an inverted index on `article_publish_date` -> `question` we can index the questions and filter out the range required, across train, val, test.

The interface is for instance equivalent to
```sql
select
*
from
questions
join
question_article_deps
on
question_id
where
t_1 < article_dep_publish_ts <= t_2
```

Also, downloading segments from the original dataset would be nice.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.