Considerations around label-based indexing for time-series data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
ScippNexus has supported label-based indexing for groups such as NXlog and NXevent_data for a while. When trying to use this, I have come across two problems:
- Sometimes data is not perfectly ordered, e.g., if the data from one pulse is slightly delayed. This fails to load with the current mechanism, since Scipp label-based indexing required ordered coords. See also scipp/scipp#3527
- Given an interval, Scipp will load all data points in between. This is not what we commonly need from NXlogs — we are more interested in the previous value (or at least need to include it). Otherwise we might end up loading an empty range.
It is not clear if 2.) should simply be solved after loading, e.g., by loading from the start. Or is there a conceptually better mechanism that could be implemented in ScippNexus?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing label-based indexing paths for NXlog and NXevent_data, then reproduce the two reported cases: slightly unordered data and an interval with no points between its bounds. Compare the behavior with scipp/scipp#3527 and determine a mechanism that includes the preceding value; done means the approach is agreed and covered by tests for both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100