reading "lists" of events into numpy arrays
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 35
- Forks
- 14
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 1
Description
Disclaimer: I am not sure if the following is compatible with the way yardl is designed.
Another way to potentially improve the efficiency of the python-based reader for "lists" of events with fixed (uncompressed) structure would be the following:
- assume we have an event with a fixed structure (e.g. 1 float, 2 ints, or e.g. a 32bit word that encodes 4x 6bit unints, and 2x 4bit ints, ...)
- assume that we stream and store many of those event (1e6) into a "TimeBlock"
If the structure of the events is fixed, numpy's memmap together with a custom dtype could be used to read all events efficiently in the TimeBlock into a "2D" array avoiding more expensive loops over events.
Obviously, we don't want to loose the compression feature, but instead of compressing individual events, the whole TimeBlock could be compressed / decompressed.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Python-based reader and the TimeBlock handling mentioned in the issue; no specific files or tests are named. Determine how fixed-structure event lists and compression are currently handled, then define measurable completion criteria around efficient array reads while preserving the compression feature.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100