awslabs / awslabs/sagemaker-debugger

Logging error: I/O operation on closed file

Open
#270 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
165
Forks
82
PR merge metrics
No merged PRs in 30d

Description

Observing this at the end of a run of `pytest tests/tensorflow2`. Issue most probably introduced after PR https://github.com/awslabs/sagemaker-debugger/pull/259

```
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.6/logging/__init__.py", line 996, in emit
stream.write(msg)
File "/usr/local/lib/python3.6/dist-packages/_pytest/capture.py", line 424, in write
return self.buffer.write(s.encode(self.encoding, "replace"))
ValueError: I/O operation on closed file
Call stack:
File "/usr/local/lib/python3.6/dist-packages/smdebug/tensorflow/keras.py", line 695, in _cleanup
super()._cleanup()
File "/usr/local/lib/python3.6/dist-packages/smdebug/core/hook.py", line 500, in _cleanup
self._close_writers()
File "/usr/local/lib/python3.6/dist-packages/smdebug/tensorflow/base_hook.py", line 349, in _close_writers
self._write_scalars()
File "/usr/local/lib/python3.6/dist-packages/smdebug/core/hook.py", line 648, in _write_scalars
if self._is_not_supported():
File "/usr/local/lib/python3.6/dist-packages/smdebug/tensorflow/keras.py", line 68, in _is_not_supported
"Executing in TF2.x eager mode."
Message: 'Executing in TF2.x eager mode.SageMaker Debugger will not be saving gradients'
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with `pytest tests/tensorflow2`, then trace the cleanup path through `smdebug/tensorflow/keras.py`, `smdebug/core/hook.py`, and `smdebug/tensorflow/base_hook.py`. Check the logging call in `_is_not_supported` and the writer-closing flow; done means the TensorFlow 2 test run finishes without an I/O-on-closed-file logging error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.