google-deepmind / google-deepmind/streamingqa
Tool for splitting dataset by `publish_date` range?
Open
- Dominant language
- Python
- Stars
- 51
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
It would be nice to have a tool for splitting up the dataset into any arbitrary `t=t_1 to t=t_2`.
For instance, if we index the corpus along the publish date, then created an inverted index on `article_publish_date` -> `question` we can index the questions and filter out the range required, across train, val, test.
The interface is for instance equivalent to
```sql
select
*
from
questions
join
question_article_deps
on
question_id
where
t_1 < article_dep_publish_ts <= t_2
```
Also, downloading segments from the original dataset would be nice.
Contributor guide
Assessment
This issue has not been assessed yet.