tensorflow / tensorflow/datasets
[GSoC] Add a dummy beam wrapper
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 1.6k
- Avg merge
- 3h 54m
- Merged PRs (30d)
- 1
Description
Currently, implementing a beam dataset is quire painful due to lazy imports, which makes it impossible to use beam.DoFn,... in the global scope.
It would be nice to be able to implement beam datasets without worrying about declaring beam methods only in the main scope.
By replacing the beam module by a dummy no-op module, it would not crash when importing tensorflow datasets. It would greatly improve the usability:
try:
import appache_beam as beam
except ImportError:
beam = DummyBeam
class SomeFn(beam.DoFn):
pass
@beam.ptransform_fn
@beam.typehints.with_input_types(beam.Pipeline)
@beam.typehints.with_output_types(Union[tf.train.Example, bytes])
def some_ptransform_fn():
pass
- If beam is installed, everything works as expected.
- If beam isn't found, then the imports works, but calling
SomeFn(),some_ptransform_fn(),... raise the lazy import error ('Please install appache_beam')
Once https://github.com/tensorflow/datasets/issues/1945 is done, the try/except could be replaced by:
with tfds.core.lazy_imports():
import appache_beam as beam
class SomeFn(beam.DoFn):
pass
@beam.ptransform_fn
def some_ptransform_fn():
pass
Note: Not all the beam API has to be implemented. New methods can be added on the go as they are needed.
The implementation could go into /core/utils/dummy_beam.py, and tests in /core/utils/dummy_beam_test.py.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read the lazy-import behavior used by TensorFlow Datasets, then inspect the proposed paths /core/utils/dummy_beam.py and /core/utils/dummy_beam_test.py. Define the Beam API surface needed by the issue examples and verify that imports succeed without Apache Beam while calls raise the lazy import error. Done means the documented wrapper behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100