Silent Pagination Truncation when using `--page-size` or `--starting-token` on affected services
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
### Describe the bug
There is a silent data truncation bug in the AWS CLI's automatic pagination logic. When users run a paginated command that happens to have a native API `limit_key` named `"PageSize"` (e.g., `aws elbv2 describe-load-balancers`), explicitly using the global `--page-size` argument will **silently disable automatic pagination**. The CLI will fetch only the first page and exit with a `0` exit code, giving the user the false impression that they have fetched all resources in batches.
Similarly, if an API's `input_token` is exactly `"StartingToken"` (e.g., `aws ssm-quicksetup list-configurations`), using the global `--starting-token` argument will also silently disable automatic pagination.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Expected Behavior
When using the global `--page-size` flag, the CLI should automatically fetch all items across all pages in batches of the specified size. It should not stop after the first page unless `--no-paginate` or `--max-items` is specified.
### Current Behavior
When providing `--page-size` to an operation whose underlying API natively uses `PageSize` as its pagination limit parameter, the AWS CLI mistakenly identifies the global argument as a manual, undocumented API parameter and disables automatic pagination. It retrieves exactly one page (of size equal to `--page-size`) and exits with code `0`.
There are no errors, uncaught exceptions, or stack traces—it fails silently.
### Reproduction Steps
Observe the difference in results when using `--page-size`:
```bash
# This fetches ALL load balancers automatically via pagination.
aws elbv2 describe-load-balancers
# EXPECTED: Fetches ALL load balancers automatically, but in batches of 1.
# ACTUAL: Fetches EXACTLY 1 load balancer, silently disables pagination, and exits with code 0!
aws elbv2 describe-load-balancers --page-size 1
```
### Possible Solution
The issue is caused by a typographical error and a missing list element in `awscli/customizations/paginate.py` within the `check_should_enable_pagination` function:
```python
normalized_paging_args = ['start_token', 'max_items']
```
1. **Typo:** `start_token` is used instead of the correct `starting_token`.
2. **Omission:** `page_size` is completely missing from `normalized_paging_args`.
Fix the tuple to include both correctly, matching the `paging_params` list defined slightly lower in the same file:
```diff
- normalized_paging_args = ['start_token', 'max_items']
+ normalized_paging_args = ['starting_token', 'page_size', 'max_items']
```
### Additional Information/Context
The following API operations are affected and will suffer from silent truncation when `--page-size` is provided:
- `ce` (Cost Explorer): `GetReservationPurchaseRecommendation`, `GetRightsizingRecommendation`, `ListCommitmentPurchaseAnalyses`, `ListSavingsPlansPurchaseRecommendationGeneration`
- `elb` / `elbv2`: `DescribeLoadBalancers`, `DescribeAccountLimits`, `DescribeTargetGroups`, `DescribeListeners`, `DescribeRules`, `DescribeSSLPolicies`, `DescribeTrustStores`, etc.
- `lakeformation`: `GetWorkUnits`
- `mailmanager`: `ListAddonInstances`, `ListArchives`, `ListRuleSets`, `ListTrafficPolicies`, etc.
- `pinpoint-email`: `GetDedicatedIps`, `ListConfigurationSets`, `ListEmailIdentities`, etc.
- `servicecatalog`: `SearchProductsAsAdmin`, `ListPortfolios`, `ListProvisionedProductPlans`, `ScanProvisionedProducts`, etc.
- `sesv2`: `ListMultiRegionEndpoints`, `ListTenants`, etc.
- `ssm-quicksetup`: `ListConfigurations` and `ListConfigurationManagers` (affected by `--starting-token` truncation).
This bug is particularly insidious for users operating large environments who explicitly use `--page-size` to prevent timeouts when retrieving thousands of ELBs or resources.
### CLI version used
aws-cli/1.x.x and aws-cli/2.x.x (Affects all recent v1 and v2 versions since the pagination customization was originally implemented)
### Environment details (OS name and version, etc.)
All environments and operating systems (Linux, macOS, Windows)
Contributor guide
Assessment
This issue has not been assessed yet.