awslabs / awslabs/amazon-dynamodb-tools

[bulk] Client prints a urllib3 traceback after stopping a job

Open Beginner friendly
#353 1 comment 0 reactions 0 assignees View on GitHub
bulk_executor help wanted
Dominant language
Python
Stars
181
Forks
50
Avg merge
4h 11m
Merged PRs (30d)
36

Description

## What happens

When bulk stops a Glue job because the logs said it was doomed, the run sometimes ends with
a traceback from our own process, after the closing line:

```
2026-09-01 17:30:17,521 WARNING - Job was stopped. Job duration: 0:02:47
Exception ignored in:
Traceback (most recent call last):
File ".../urllib3/response.py", line 1289, in close
self._fp.close()
File ".../http/client.py", line 432, in close
super().close() # set "closed" flag
File ".../http/client.py", line 445, in flush
self.fp.flush()
ValueError: I/O operation on closed file.
```

Measured on run `jr_30e6d16a...` (a `sql` query stopped after an executor ran out of
memory). It is the tail end of the CloudWatch Live Tail event stream being garbage-collected
after `event_stream.close()`, so it is harmless -- but it is the last thing on screen, it is
a traceback, and it names files in our virtualenv. A user who has just been told their job
failed now has to decide whether this is part of the failure. It is not.

"Exception ignored in" means Python is reporting an exception raised inside `__del__`, which
cannot propagate, so nothing downstream is affected and the exit code is unchanged.

## Fix

Close the live-tail stream deterministically rather than leaving it to the collector -- the
close path in `_watch_log_group` already calls `event_stream.close()` in some branches but
not on the unhealthy-shutdown return, which is exactly the path where this was seen.

Not reproducible on every stop, which suggests it depends on when the collector runs relative
to interpreter shutdown; a test would need to drive the close path rather than assert on
output.

Contributor guide

Open the contributing guide

Research direction

Locate the bulk client's _watch_log_group implementation and trace the unhealthy-shutdown return, comparing it with branches that already call event_stream.close(). Drive that close path in a test, including the unhealthy shutdown, and confirm the live-tail stream is closed deterministically without the ignored urllib3 traceback.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
observability
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.