aws / aws/aws-sam-cli

sam local invoke leaks Docker container and temp directory when function is OOM-killed

Open Beginner friendly
#9,182 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.7k
Forks
1.2k
Avg merge
1d 10h
Merged PRs (30d)
52

Description

### Description

When a function invoked via `sam local invoke` (or `sam local start-api`/`start-lambda`) exceeds its `MemorySize` and is OOM-killed, the resulting `ContainerFailureError` is raised *before* the container is stopped/removed and before the per-invocation decompressed-code temp directory is cleaned up. Both are leaked.

### Root cause

```python
# samcli/local/lambdafn/runtime.py:350-362
def _on_invoke_done(self, container):
if container:
self._check_exit_state(container) # <-- raises ContainerFailureError on OOM
self._container_manager.stop(container) # <-- never reached
self._clean_decompressed_paths() # <-- never reached
```

`_on_invoke_done` is invoked unconditionally from `invoke()`'s `finally` block:

```python
# samcli/local/lambdafn/runtime.py:343-346
finally:
# We will be done with execution, if either the execution completed or an interrupt was fired
# Any case, cleanup the container.
self._on_invoke_done(container)
```

`_on_invoke_done` has no try/except of its own, so when `_check_exit_state()` raises `ContainerFailureError` on OOM (`runtime.py:364-382`), that exception propagates immediately — skipping both `self._container_manager.stop(container)` and `self._clean_decompressed_paths()` on the same line and below it.

### Steps to reproduce

1. Create a function with a low `MemorySize` (e.g. 128) that allocates memory past the limit in its handler.
2. Run `sam local invoke` against it.
3. Observe the invocation fails with a memory error as expected, but:
- `docker ps -a` shows the container still present (not stopped/removed).
- The temp directory created by `_unzip_file`/`_get_code_dir` for that invocation is not removed from disk.
4. Repeating the invocation accumulates one leaked container and one leaked temp directory per failed run.

### Expected result
Regardless of why `_check_exit_state` raises, the container should still be stopped/removed and the temp directory cleaned up — cleanup should not be skippable by an exception raised partway through `_on_invoke_done`.

### Suggested fix
Wrap the cleanup steps so they always run regardless of `_check_exit_state`'s outcome, e.g.:

```python
def _on_invoke_done(self, container):
try:
if container:
self._check_exit_state(container)
finally:
if container:
self._container_manager.stop(container)
self._clean_decompressed_paths()
```
(or equivalent try/finally ordering that guarantees `stop()` and `_clean_decompressed_paths()` run before/regardless of re-raising the OOM error.)

### Impact
Repeated local-invoke testing against a memory-constrained function accumulates dead Docker containers and temp directories over a test loop — not catastrophic, but a real, persistent leak that requires manual `docker container prune` / temp-dir cleanup, and gets worse the more a developer iterates on tuning `MemorySize` locally (the exact workflow most likely to repeatedly trigger this).

### Environment
- `aws-sam-cli` version: 1.165.0 (current `develop` branch, commit at time of testing)
- OS: macOS

Contributor guide

Open the contributing guide

Research direction

Start in samcli/local/lambdafn/runtime.py at invoke() and _on_invoke_done(), then inspect _check_exit_state(), _container_manager.stop(), and _clean_decompressed_paths(). Reproduce an OOM invocation if possible and verify that the ContainerFailureError still appears while the container and decompressed-code temp directory are cleaned up.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
cli
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.