DiskDataset assumes the X field is populated, in several places
- Dominant language
- Python
- Stars
- 7k
- Forks
- 2.3k
- PR merge metrics
- No merged PRs in 30d
Description
While it is possible to easily create a DiskDataset that does not have the X field populated, there are lots of places in the DiskDataset code that assume that it is.
There is a strong use case for being able to make DiskDataset objects with, say, just a y field. For doing inference on large datasets, it makes sense to create a DiskDataset to contain the resulting generated data, and the most reasonable place to put this is in the y field.
There are several places in the code this assumption is made; for example in get_shard() it's assumed that the X file is there. Notably, this is not checked for on DiskDataset creation.
Contributor guide
Research direction
Begin at DiskDataset creation and get_shard(), then locate the other DiskDataset paths that assume an X file exists. Done means a DiskDataset containing only y can be created and used through those paths without requiring X, with regression coverage for the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100