aws / aws/aws-cli

Bug Report: `aws --output text` with `--query` prints a spurious `None` line for the pagination resume-token hint

Open
#10,449 2 comments 0 reactions 1 assignee Claimed by @aemous View on GitHub
bug community contribution-ready p3 paginator
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the bug

When `--output text` is combined with `--query` and a paginated response is
**truncated** (for example via `--max-items`), the AWS CLI appends a stray
`None` line to the output.

After the last page, the text formatter prints a `NEXTTOKEN` "resume token"
hint so users know how to continue paginating. That hint is pagination
*metadata*, but it is incorrectly run through the user's `--query` expression.
Evaluating a data expression such as `Users[].UserName` against the hint object
`{"NextToken": {"NextToken": ""}}` returns `None`, and the text formatter
writes the literal string `None`.

In scripts that consume `--output text --query` line by line, the extra `None`
silently corrupts the data (an additional row, a wrong count, or a bogus field
value).

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

The resume-token hint should render exactly as it does **without** `--query`,
because it is not part of the queried result set:

```
alice
NEXTTOKEN
```

### Current Behavior

```
alice
None
```

The trailing `None` is the result of applying `Users[].UserName` to the
resume-token hint object instead of to the page data.

### Reproduction Steps

```bash
aws iam list-users \
--max-items 1 \
--output text \
--query 'Users[].UserName'
```

Any paginated operation reproduces this; `--max-items` is simply the easiest
way to force a truncated response (and therefore a resume token). The bug
appears whenever **all** of the following hold:

1. output format is `text`,
2. `--query` is supplied, and
3. the response is truncated, so a resume token is generated.

Minimal, self-contained reproduction against mocked responses (no AWS account
or network required — mirrors the regression test):

```python
from awscli.testutils import BaseAWSCommandParamsTest

class Repro(BaseAWSCommandParamsTest):
def test_repro(self):
self.parsed_responses = [
{'Users': [{'UserName': 'alice', 'Path': '/',
'CreateDate': '2024-01-01T00:00:00Z',
'UserId': 'A1', 'Arn': 'arn:aws:iam::12345:user/alice'}],
'IsTruncated': True, 'Marker': 'PAGE2'},
{'Users': [{'UserName': 'bob', 'Path': '/',
'CreateDate': '2024-01-02T00:00:00Z',
'UserId': 'B1', 'Arn': 'arn:aws:iam::12345:user/bob'}],
'IsTruncated': False},
]
out = self.run_cmd(
'iam list-users --max-items 1 --output text '
'--query Users[].UserName', expected_rc=0)[0]
# Actual: 'alice\nNone\n'
# Expected: 'alice\nNEXTTOKEN\t\n'
print(repr(out))
```

### Possible Solution

The bug is in `awscli/formatter.py`, `TextFormatter.__call__`:

```python
if response.resume_token:
# Tell the user about the next token so they can continue
# if they want.
self._format_response(
{'NextToken': {'NextToken': response.resume_token}}, stream)
```

`_format_response` applies `--query` to whatever it is given:

```python
def _format_response(self, response, stream):
if self._args.query is not None:
response = self._args.query.search(response)
text.format_text(response, stream)
```

Render the resume-token hint directly with `text.format_text(...)`, bypassing
`--query`, so the hint is preserved for all output cases:

```python
if response.resume_token:
text.format_text(
{'NextToken': {'NextToken': response.resume_token}}, stream)
```

Only `TextFormatter` is affected. `FullyBufferedFormatter` (JSON and Table)
aggregates the full result with `build_full_result()` and applies `--query`
once to the entire response, so it has no separate hint emission and does not
exhibit this bug.

### Additional Information/Context

- Affects every paginated operation, not just `iam list-users`.
- The hint is only emitted when a resume token exists, i.e. when the response
is truncated (commonly `--max-items`), so non-truncated commands are
unaffected.
- Not a regression — the behavior has existed since the text formatter began
streaming pages.

### CLI version used

`aws-cli/1.45.36` (reproduced on the `develop` branch). The same code path exists in the v2 line.

### Environment details (OS name and version, etc.)

Platform independent (verified on Linux); no platform-specific behavior is involved.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.