microsoft / microsoft/yardl

reading "lists" of events into numpy arrays

Open
#230 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
35
Forks
14
Avg merge
5d 17h
Merged PRs (30d)
1

Description

Disclaimer: I am not sure if the following is compatible with the way yardl is designed.

Another way to potentially improve the efficiency of the python-based reader for "lists" of events with fixed (uncompressed) structure would be the following:

  • assume we have an event with a fixed structure (e.g. 1 float, 2 ints, or e.g. a 32bit word that encodes 4x 6bit unints, and 2x 4bit ints, ...)
  • assume that we stream and store many of those event (1e6) into a "TimeBlock"

If the structure of the events is fixed, numpy's memmap together with a custom dtype could be used to read all events efficiently in the TimeBlock into a "2D" array avoiding more expensive loops over events.

Obviously, we don't want to loose the compression feature, but instead of compressing individual events, the whole TimeBlock could be compressed / decompressed.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Python-based reader and the TimeBlock handling mentioned in the issue; no specific files or tests are named. Determine how fixed-structure event lists and compression are currently handled, then define measurable completion criteria around efficient array reads while preserving the compression feature.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.