aws / aws/aws-cli

`cloudtrail validate-logs`: parallelize log file download and hash verification

Open
#10,516 0 comments 1 reaction 1 assignee Claimed by @RyanFitzSimmonsAK View on GitHub
cloudtrail feature-request investigating p3
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the feature

`aws cloudtrail validate-logs` currently downloads and verifies every log file referenced by each digest file **sequentially**. In `awscli/customizations/cloudtrail/validation.py`, `CloudTrailValidateLogs._call()` iterates digests and calls `self._download_log(log)` one file at a time (one `GetObject` + SHA-256 per file, single-threaded).

CloudTrail delivers many small gzipped log files (typically one every ~5 minutes per region), so total runtime is dominated by `number of log files × S3 round-trip latency`. The digest chain signature verification is inherently sequential (each digest references the previous one via `previousDigestSignature`), but the per-digest log file hash verification is embarrassingly parallel: each log file's SHA-256 is independently compared against the hash recorded in its digest.

I propose parallelizing the log file download + hash verification step (e.g. with a bounded `concurrent.futures.ThreadPoolExecutor`), while keeping the digest chain traversal and signature verification sequential as designed.

### Use Case

We run a scheduled organization-wide integrity check (organization trail, many member accounts × many regions) in CI. Because the digest chains are independent per account × region, we already fan out at the account level, but each `validate-logs` invocation still takes a long time because every referenced log file is fetched one by one. For a 24h window on active accounts this means thousands of sequential S3 GETs per region. Parallelizing the hash verification step would reduce wall-clock time roughly by the concurrency factor, since the workload is almost entirely network-bound.

### Proposed Solution

- Keep `DigestTraverser.traverse_digests()` (chain walking + RSA signature verification) sequential — required by the `previousDigestSignature` chain design.
- In `CloudTrailValidateLogs._call()`, replace the inner `for log in digest['logFiles']: self._download_log(log)` loop with a bounded thread pool (workers configurable, conservative default such as 8–10) that downloads and hashes log files concurrently.
- `hashlib.sha256` releases the GIL and the work is I/O-bound, so threads are sufficient; no multiprocessing needed.
- Buffer/serialize status output per file so `--verbose` output and the valid/invalid counters remain deterministic and thread-safe (counters updated under a lock or aggregated from futures).
- Exit code semantics unchanged: any invalid log file still results in a non-zero exit.
- Could be opt-in via a new option (e.g. `--concurrent-requests `) if changing the default behavior is a concern; output ordering within a digest would be the only observable difference.

I'm willing to work on a PR for this if the approach sounds acceptable.

### Other Information

Related: #4250 (validate-logs spends a long time listing objects when no log files exist in range) — different bottleneck (ListObjects), but same overall pain point of validate-logs runtime.

For an organization trail, the effect multiplies across accounts and regions because each account × region has its own digest chain and therefore its own `validate-logs` invocation.

This issue was drafted with the assistance of AI tools (Claude Code), and reviewed by Kohei Kamada before submission.

### Acknowledgements

- [x] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### CLI version used

aws-cli/2.11.6 Python/3.11.2 Darwin/25.5.0 exe/x86_64 (behavior confirmed unchanged on current `develop` branch of aws/aws-cli)

### Environment details (OS name and version, etc.)

macOS 26.5.2 (Darwin 25.5.0), x86_64; also reproducible on Ubuntu GitHub Actions runners

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.