lasp / lasp/space_packet_parser

Support Multiprocessing in create_dataset

Open
#241 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
39
Forks
15
Avg merge
2d 16h
Merged PRs (30d)
12

Description

Context

When working with muxed packet files generated from ground testing data, we often have to inefficiently parse thousands of files, filtering for the APID of interest. This is quite slow when create_dataset parses each file serially.

Driving Requirements

Support an optional n_workers kwarg to create_dataset to enable multiprocessing of input packet files.

Implementation Requirements

To save the overhead of creating many processes, multiprocessing inside create_dataset should probably spin up n_worker processes (or n_files processes if n_files < n_workers) and send roughly an equal number of packet files to each worker process.

This parallelization should apply only to the parsing loop itself with all the post processing for numpy data types occurring afterwards.

Considerations

Make sure we aren't forgetting about any consistency checking that occurs in the packet parsing loop (e.g. duplicates or anything like that). I don't think this will be an issue but good to think about.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the existing create_dataset entry point and trace its serial packet-file parsing loop, including any duplicate or other consistency checks. Implement the optional n_workers behavior around parsing only, preserving the later NumPy post-processing. Done means files are divided across no more than the requested worker count and parsing results remain consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.