tensorflow / tensorflow/datasets

[GSoC] Add a dummy beam wrapper

Open
#1,946 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

contributions welcome enhancement
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

Currently, implementing a beam dataset is quire painful due to lazy imports, which makes it impossible to use beam.DoFn,... in the global scope.
It would be nice to be able to implement beam datasets without worrying about declaring beam methods only in the main scope.

By replacing the beam module by a dummy no-op module, it would not crash when importing tensorflow datasets. It would greatly improve the usability:

try:
  import appache_beam as beam
except ImportError:
  beam = DummyBeam


class SomeFn(beam.DoFn):
  pass


@beam.ptransform_fn
@beam.typehints.with_input_types(beam.Pipeline)
@beam.typehints.with_output_types(Union[tf.train.Example, bytes])
def some_ptransform_fn():
  pass
  • If beam is installed, everything works as expected.
  • If beam isn't found, then the imports works, but calling SomeFn(), some_ptransform_fn(),... raise the lazy import error ('Please install appache_beam')

Once https://github.com/tensorflow/datasets/issues/1945 is done, the try/except could be replaced by:

with tfds.core.lazy_imports():
  import appache_beam as beam


class SomeFn(beam.DoFn):
  pass


@beam.ptransform_fn
def some_ptransform_fn():
  pass

Note: Not all the beam API has to be implemented. New methods can be added on the go as they are needed.

The implementation could go into /core/utils/dummy_beam.py, and tests in /core/utils/dummy_beam_test.py.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the lazy-import behavior used by TensorFlow Datasets, then inspect the proposed paths /core/utils/dummy_beam.py and /core/utils/dummy_beam_test.py. Define the Beam API surface needed by the issue examples and verify that imports succeed without Apache Beam while calls raise the lazy import error. Done means the documented wrapper behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.