Bug Report: `aws --output text` with `--query` prints a spurious `None` line for the pagination resume-token hint
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
### Describe the bug
When `--output text` is combined with `--query` and a paginated response is
**truncated** (for example via `--max-items`), the AWS CLI appends a stray
`None` line to the output.
After the last page, the text formatter prints a `NEXTTOKEN` "resume token"
hint so users know how to continue paginating. That hint is pagination
*metadata*, but it is incorrectly run through the user's `--query` expression.
Evaluating a data expression such as `Users[].UserName` against the hint object
`{"NextToken": {"NextToken": ""}}` returns `None`, and the text formatter
writes the literal string `None`.
In scripts that consume `--output text --query` line by line, the extra `None`
silently corrupts the data (an additional row, a wrong count, or a bogus field
value).
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Expected Behavior
The resume-token hint should render exactly as it does **without** `--query`,
because it is not part of the queried result set:
```
alice
NEXTTOKEN
```
### Current Behavior
```
alice
None
```
The trailing `None` is the result of applying `Users[].UserName` to the
resume-token hint object instead of to the page data.
### Reproduction Steps
```bash
aws iam list-users \
--max-items 1 \
--output text \
--query 'Users[].UserName'
```
Any paginated operation reproduces this; `--max-items` is simply the easiest
way to force a truncated response (and therefore a resume token). The bug
appears whenever **all** of the following hold:
1. output format is `text`,
2. `--query` is supplied, and
3. the response is truncated, so a resume token is generated.
Minimal, self-contained reproduction against mocked responses (no AWS account
or network required — mirrors the regression test):
```python
from awscli.testutils import BaseAWSCommandParamsTest
class Repro(BaseAWSCommandParamsTest):
def test_repro(self):
self.parsed_responses = [
{'Users': [{'UserName': 'alice', 'Path': '/',
'CreateDate': '2024-01-01T00:00:00Z',
'UserId': 'A1', 'Arn': 'arn:aws:iam::12345:user/alice'}],
'IsTruncated': True, 'Marker': 'PAGE2'},
{'Users': [{'UserName': 'bob', 'Path': '/',
'CreateDate': '2024-01-02T00:00:00Z',
'UserId': 'B1', 'Arn': 'arn:aws:iam::12345:user/bob'}],
'IsTruncated': False},
]
out = self.run_cmd(
'iam list-users --max-items 1 --output text '
'--query Users[].UserName', expected_rc=0)[0]
# Actual: 'alice\nNone\n'
# Expected: 'alice\nNEXTTOKEN\t\n'
print(repr(out))
```
### Possible Solution
The bug is in `awscli/formatter.py`, `TextFormatter.__call__`:
```python
if response.resume_token:
# Tell the user about the next token so they can continue
# if they want.
self._format_response(
{'NextToken': {'NextToken': response.resume_token}}, stream)
```
`_format_response` applies `--query` to whatever it is given:
```python
def _format_response(self, response, stream):
if self._args.query is not None:
response = self._args.query.search(response)
text.format_text(response, stream)
```
Render the resume-token hint directly with `text.format_text(...)`, bypassing
`--query`, so the hint is preserved for all output cases:
```python
if response.resume_token:
text.format_text(
{'NextToken': {'NextToken': response.resume_token}}, stream)
```
Only `TextFormatter` is affected. `FullyBufferedFormatter` (JSON and Table)
aggregates the full result with `build_full_result()` and applies `--query`
once to the entire response, so it has no separate hint emission and does not
exhibit this bug.
### Additional Information/Context
- Affects every paginated operation, not just `iam list-users`.
- The hint is only emitted when a resume token exists, i.e. when the response
is truncated (commonly `--max-items`), so non-truncated commands are
unaffected.
- Not a regression — the behavior has existed since the text formatter began
streaming pages.
### CLI version used
`aws-cli/1.45.36` (reproduced on the `develop` branch). The same code path exists in the v2 line.
### Environment details (OS name and version, etc.)
Platform independent (verified on Linux); no platform-specific behavior is involved.
Contributor guide
Assessment
This issue has not been assessed yet.