Handling dask array with unknown dimensions
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 12
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
When the output shape of an operation is unknown, the output is still a dask array whose shape is treated as nan (not a number). When we try to convert this array of unknown shape into a sidpy dataset, it raises an error.
For example,
dset = sid.Dataset.from_array(np.random.rand(4,5))
new_dset = dset[dset<0.5] # The shape of new_dset is unknown until we use .compute() on it.
The shape of new_dset is (nan,) and dset.like_data(new_dset) does not work. This is important when modifying getitem() to always return a sidpy dataset instead of a dask array.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the example with Dataset.from_array, boolean indexing through getitem(), and like_data() on the resulting unknown-shape array. Trace how the (nan,) shape is handled before conversion, then verify that the dataset conversion succeeds without computing the array and remains compatible with the planned getitem() behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100