tensorflow / tensorflow/tensorboard

Loading data on HDFS with reduced feature set (w/o TensorFlow installation)

Open
#6,165 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

When loading TensorBoard data on HDFS without TensorFlow installation, I got the following error log (note: I modified HDFS data paths in the log) and could not load the data. Recently dependent module fsspec has updated (fsspec==2023.1.0) and supported seeking API

In response, I modified tensorboard/compat/tensorflow_stub/io/gfile.py as like
https://github.com/nyamashi/tensorboard/commit/ea9f6e6dd3c36b19d51563f2ab2f0ef7f87e3d40.
Then it looks working.

Is it possible to enhance the dependency (fsspec >= 2023.1.0) and add this modification in tensorboard/compat/tensorflow_stub/io/gfile.py?

Error message before the modification:

$ tensorboard --inspec --logdir {sample hdfs path}
TensorFlow installation not found - running with reduced feature set.
======================================================================
Processing event files... (this can take a few minutes)
======================================================================

Traceback (most recent call last):
  File "/opt/myenv/python38/bin/tensorboard", line 8, in <module>
    sys.exit(run_main())
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/main.py", line 46, in run_main
    app.run(tensorboard.main, flags_parser=tensorboard.configure)
  File "/opt/myenv/python38/lib/python3.8/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/opt/myenv/python38/lib/python3.8/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/program.py", line 276, in main
    return runner(self.flags) or 0
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/program.py", line 289, in _run_serve_subcommand
    efi.inspect(flags.logdir, event_file, flags.tag)
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/backend/event_processing/event_file_inspector.py", line 430, in inspect
    inspection_units = get_inspection_units(logdir, event_file, tag)
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/backend/event_processing/event_file_inspector.py", line 384, in get_inspection_units
    field_to_obs=get_field_to_observations_map(generator, tag),
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/backend/event_processing/event_file_inspector.py", line 191, in get_field_to_observations_map
    for event in generator:
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 244, in Load
    for record in super(LegacyEventFileLoader, self).Load():
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 178, in Load
    yield next(self._iterator)
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 109, in __next__
    self._reader.GetNext()
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py", line 207, in GetNext
    header_str = self._read(8)
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py", line 273, in _read
    new_data = self.file_handle.read(n)
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 728, in read
    (self.buff, self.continuation_token) = self.fs.read(
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 447, in func_wrapper
    return func(self, *args, **kwargs)
  File "/opt/myenv/python38/lib/python3.8/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 526, in read
    raise errors.InvalidArgumentError(
tensorboard.compat.tensorflow_stub.errors.InvalidArgumentError: b'{sample 
 hdfs path}/events.out.tfevents.xxxxx' is not seekable

With fsspec == 2023.1.0 and https://github.com/nyamashi/tensorboard/commit/ea9f6e6dd3c36b19d51563f2ab2f0ef7f87e3d40. Looks working:

$ tensorboard --inspec --logdir  {sample hdfs path}
TensorFlow installation not found - running with reduced feature set.
======================================================================
Processing event files... (this can take a few minutes)
======================================================================

Found event files in:
 {sample hdfs path}

These tags are in  {sample hdfs path}
audio -
histograms -
images -
scalars
   .....
tensor
   ....
======================================================================

Event statistics for  {sample hdfs path}
audio -
graph -
histograms -
images -
scalars
   ....
sessionlog:checkpoint -
sessionlog:start -
sessionlog:stop -
tensor
   ....
======================================================================

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tensorboard/compat/tensorflow_stub/io/gfile.py and the dependency configuration for fsspec. Reproduce the reduced-feature TensorBoard command against an HDFS logdir, then verify that event inspection completes without the not-seekable error after the dependency and gfile changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.