tensorflow / tensorflow/datasets
Make register_checksums works with Beam datasets
Open
Nobody has claimed this yet.
contributions welcome
enhancement
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Currently, when dl_manager is used inside a Beam pipeline, checksums registration is disabled.
In theory, each instance of beam workers should register their checksums and save them inside some shared directory, then once all pipeline are complete, the checksums from the workers are merged together and saved.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how dl_manager is used inside Beam pipelines and where checksum registration is disabled. Review how worker outputs could be stored in a shared directory and merged after pipeline completion; done means checksums from Beam workers are registered and saved together.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100