awslabs / awslabs/sagemaker-debugger
Logging error: I/O operation on closed file
- Dominant language
- Python
- Stars
- 165
- Forks
- 82
- PR merge metrics
- No merged PRs in 30d
Description
Observing this at the end of a run of `pytest tests/tensorflow2`. Issue most probably introduced after PR https://github.com/awslabs/sagemaker-debugger/pull/259
```
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.6/logging/__init__.py", line 996, in emit
stream.write(msg)
File "/usr/local/lib/python3.6/dist-packages/_pytest/capture.py", line 424, in write
return self.buffer.write(s.encode(self.encoding, "replace"))
ValueError: I/O operation on closed file
Call stack:
File "/usr/local/lib/python3.6/dist-packages/smdebug/tensorflow/keras.py", line 695, in _cleanup
super()._cleanup()
File "/usr/local/lib/python3.6/dist-packages/smdebug/core/hook.py", line 500, in _cleanup
self._close_writers()
File "/usr/local/lib/python3.6/dist-packages/smdebug/tensorflow/base_hook.py", line 349, in _close_writers
self._write_scalars()
File "/usr/local/lib/python3.6/dist-packages/smdebug/core/hook.py", line 648, in _write_scalars
if self._is_not_supported():
File "/usr/local/lib/python3.6/dist-packages/smdebug/tensorflow/keras.py", line 68, in _is_not_supported
"Executing in TF2.x eager mode."
Message: 'Executing in TF2.x eager mode.SageMaker Debugger will not be saving gradients'
```
Contributor guide
Research direction
Reproduce the issue with `pytest tests/tensorflow2`, then trace the cleanup path through `smdebug/tensorflow/keras.py`, `smdebug/core/hook.py`, and `smdebug/tensorflow/base_hook.py`. Check the logging call in `_is_not_supported` and the writer-closing flow; done means the TensorFlow 2 test run finishes without an I/O-on-closed-file logging error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100