lasp / lasp/space_packet_parser
Support Multiprocessing in create_dataset
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39
- Forks
- 15
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 12
Description
Context
When working with muxed packet files generated from ground testing data, we often have to inefficiently parse thousands of files, filtering for the APID of interest. This is quite slow when create_dataset parses each file serially.
Driving Requirements
Support an optional n_workers kwarg to create_dataset to enable multiprocessing of input packet files.
Implementation Requirements
To save the overhead of creating many processes, multiprocessing inside create_dataset should probably spin up n_worker processes (or n_files processes if n_files < n_workers) and send roughly an equal number of packet files to each worker process.
This parallelization should apply only to the parsing loop itself with all the post processing for numpy data types occurring afterwards.
Considerations
Make sure we aren't forgetting about any consistency checking that occurs in the packet parsing loop (e.g. duplicates or anything like that). I don't think this will be an issue but good to think about.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the existing create_dataset entry point and trace its serial packet-file parsing loop, including any duplicate or other consistency checks. Implement the optional n_workers behavior around parsing only, preserving the later NumPy post-processing. Done means files are divided across no more than the requested worker count and parsing results remain consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100