aws / aws/aws-cli

Silent Pagination Truncation when using `--page-size` or `--starting-token` on affected services

Open
#10,442 1 comment 0 reactions 1 assignee Claimed by @RyanFitzSimmonsAK View on GitHub
bug needs-review p2 paginator
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the bug

There is a silent data truncation bug in the AWS CLI's automatic pagination logic. When users run a paginated command that happens to have a native API `limit_key` named `"PageSize"` (e.g., `aws elbv2 describe-load-balancers`), explicitly using the global `--page-size` argument will **silently disable automatic pagination**. The CLI will fetch only the first page and exit with a `0` exit code, giving the user the false impression that they have fetched all resources in batches.

Similarly, if an API's `input_token` is exactly `"StartingToken"` (e.g., `aws ssm-quicksetup list-configurations`), using the global `--starting-token` argument will also silently disable automatic pagination.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

When using the global `--page-size` flag, the CLI should automatically fetch all items across all pages in batches of the specified size. It should not stop after the first page unless `--no-paginate` or `--max-items` is specified.

### Current Behavior

When providing `--page-size` to an operation whose underlying API natively uses `PageSize` as its pagination limit parameter, the AWS CLI mistakenly identifies the global argument as a manual, undocumented API parameter and disables automatic pagination. It retrieves exactly one page (of size equal to `--page-size`) and exits with code `0`.

There are no errors, uncaught exceptions, or stack traces—it fails silently.

### Reproduction Steps

Observe the difference in results when using `--page-size`:

```bash
# This fetches ALL load balancers automatically via pagination.
aws elbv2 describe-load-balancers

# EXPECTED: Fetches ALL load balancers automatically, but in batches of 1.
# ACTUAL: Fetches EXACTLY 1 load balancer, silently disables pagination, and exits with code 0!
aws elbv2 describe-load-balancers --page-size 1
```

### Possible Solution

The issue is caused by a typographical error and a missing list element in `awscli/customizations/paginate.py` within the `check_should_enable_pagination` function:

```python
normalized_paging_args = ['start_token', 'max_items']
```

1. **Typo:** `start_token` is used instead of the correct `starting_token`.
2. **Omission:** `page_size` is completely missing from `normalized_paging_args`.

Fix the tuple to include both correctly, matching the `paging_params` list defined slightly lower in the same file:

```diff
- normalized_paging_args = ['start_token', 'max_items']
+ normalized_paging_args = ['starting_token', 'page_size', 'max_items']
```

### Additional Information/Context

The following API operations are affected and will suffer from silent truncation when `--page-size` is provided:

- `ce` (Cost Explorer): `GetReservationPurchaseRecommendation`, `GetRightsizingRecommendation`, `ListCommitmentPurchaseAnalyses`, `ListSavingsPlansPurchaseRecommendationGeneration`
- `elb` / `elbv2`: `DescribeLoadBalancers`, `DescribeAccountLimits`, `DescribeTargetGroups`, `DescribeListeners`, `DescribeRules`, `DescribeSSLPolicies`, `DescribeTrustStores`, etc.
- `lakeformation`: `GetWorkUnits`
- `mailmanager`: `ListAddonInstances`, `ListArchives`, `ListRuleSets`, `ListTrafficPolicies`, etc.
- `pinpoint-email`: `GetDedicatedIps`, `ListConfigurationSets`, `ListEmailIdentities`, etc.
- `servicecatalog`: `SearchProductsAsAdmin`, `ListPortfolios`, `ListProvisionedProductPlans`, `ScanProvisionedProducts`, etc.
- `sesv2`: `ListMultiRegionEndpoints`, `ListTenants`, etc.
- `ssm-quicksetup`: `ListConfigurations` and `ListConfigurationManagers` (affected by `--starting-token` truncation).

This bug is particularly insidious for users operating large environments who explicitly use `--page-size` to prevent timeouts when retrieving thousands of ELBs or resources.

### CLI version used

aws-cli/1.x.x and aws-cli/2.x.x (Affects all recent v1 and v2 versions since the pagination customization was originally implemented)

### Environment details (OS name and version, etc.)

All environments and operating systems (Linux, macOS, Windows)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.