meta-pytorch / meta-pytorch/data

Using datapipes on only parts of the input

Open
#1,030 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
179
Avg merge
6d 1h
Merged PRs (30d)
2

Description

🚀 The feature

I want to apply the functionality of a already existing datapipe to only parts of my input. Below I listed some solutions.
I would like to know the "best" solution for this problem.

Motivation, pitch

Simple example:
I have a tuple containing URLs and additional information (e.g. a text, id, ...).
I want to use the HttpReader to load images behind the URLs.
Currently the HttpReader takes URLs and yields tuples of URLs and Filestreams.
Additional information in the datapipe is not permitted.

Alternatives
  • Use Unzipper then apply the datapipe you need and then using either Zipper or IterKeyZipper zip the datapipes back together. Depending on your use-case you might also need Forker or copy a column beforehand to have your key present in both datapipes.
  • Use Mapper with input_col using a function that does what I need. If I also want to delete elements (e.g. setting skip_on_error = true in HTTPReader) I also need to add a filter. This leads to some code redundancy as the functionality already exists as a datapipe. In addition the datapipe is tested while my function is not.
  • "Add a new datapipe which accepts (source_data_datapipe, function_datapipe, input_selector, output_merge_fn)
    Each batch b of source_data_datapipe gets passed into input_selector(b) to get a processed input b' . Then b' is passed into function_datapipe for processing to get output c . Finally, output_merge_fn takes b and c and combine them into any output." While this works well with my simple example, many datapipes are not compatible with this approach. In addtition the output_merge_fn might get quite complicated depending on your use-case. "One option may be to restrict it to only work for DataPipes that do not change the cardinality of the data."
    Credits to @NivekT for coming up with this solution
  • Add input_col parameter to the Datapipes where necessary/appliccable (A lot of work and maintaining...)
Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing HttpReader, Mapper, Unzipper, Zipper, IterKeyZipper, and Forker entry points described in the issue. Compare how they handle additional fields and changes in cardinality, then determine the intended API and document its behavior for the URL-and-metadata example. Done means the project has a decided, tested approach rather than another proposed alternative.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.