aws-samples / aws-samples/sample-multi-agent-orchestration-chat-on-agentcore
Access logs bucket: add storage-class transitions and configurable retention to bound long-term log storage
- Dominant language
- TypeScript
- Stars
- 128
- Forks
- 12
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 4
Description
## Summary
The shared S3 access-logs bucket created in `AgentcoreStack` (`AccessLogsBucket`) currently retains every log object for 90 days with no storage-class transition. In a long-running deployment with steady traffic, this bucket aggregates several high-volume log sources and grows close to linearly until it reaches the 90-day steady state, which can become a non-trivial amount of Standard-class storage. This issue proposes adding storage-class transitions and making the retention period configurable so long-term log storage stays bounded and cost-efficient.
## Current behavior
`packages/cdk/lib/agentcore-stack.ts` defines a single shared access-logs bucket:
```ts
this.accessLogsBucket = new s3.Bucket(this, 'AccessLogsBucket', {
// ...
lifecycleRules: [
{
id: 'DeleteOldLogs',
expiration: cdk.Duration.days(90),
},
],
objectOwnership: s3.ObjectOwnership.OBJECT_WRITER, // required for CloudFront logging
});
```
This one bucket is the destination for multiple log producers, each under its own prefix, e.g.:
- S3 server access logs for the user-storage bucket (`user-storage/`)
- S3 server access logs and CloudFront access logs for the frontend (`frontend-s3/`, `cloudfront/`)
- Athena query output (`athena-output/`)
All objects stay in **S3 Standard** for the full 90 days before expiring.
## Why this can be a problem
- **Storage grows until the 90-day steady state.** Because objects are only deleted at 90 days and there is no transition to a cheaper class, accumulated log volume rises continuously for the first ~90 days of operation. For an actively used deployment, the combined CloudFront + S3 server access logs can reach tens of GiB of Standard-class storage before old objects begin expiring.
- **No cost taper.** Access logs are write-once / rarely-read. Keeping them all in S3 Standard for 90 days misses the obvious opportunity to transition them to a lower-cost class (e.g., Standard-IA or Glacier Instant Retrieval) well before expiration.
- **One-size-fits-all retention.** A single 90-day rule applies uniformly across very different log sources (high-volume CloudFront/S3 access logs vs. low-volume Athena output). The retention period is also hard-coded and not exposed as a stack/env configuration.
This is purely an infrastructure / cost-hygiene improvement — it is not a functional bug and does not affect application behavior.
## Proposed enhancement
1. **Add storage-class transitions** to the `AccessLogsBucket` lifecycle rule, for example:
- Transition to `INFREQUENT_ACCESS` (or `GLACIER_IR`) after ~30 days
- Keep `expiration` at the configured retention (default 90 days)
2. **Make retention configurable** via the existing env/stack configuration mechanism (e.g., `accessLogsRetentionDays`) instead of a hard-coded 90 days, with a sensible default.
3. *(Optional)* **Per-prefix policies** so high-volume prefixes (CloudFront, S3 server access logs) can use shorter retention / earlier transition than low-volume prefixes (Athena output).
4. *(Optional)* Document the expected steady-state size and provide a CloudWatch `BucketSizeBytes` alarm example in the docs so operators can monitor growth.
### Example sketch
```ts
lifecycleRules: [
{
id: 'AccessLogsLifecycle',
transitions: [
{
storageClass: s3.StorageClass.INFREQUENT_ACCESS,
transitionAfter: cdk.Duration.days(30),
},
],
expiration: cdk.Duration.days(accessLogsRetentionDays), // default 90
},
],
```
## Acceptance criteria
- [ ] Access-logs bucket lifecycle includes at least one storage-class transition before expiration.
- [ ] Retention period is configurable (with a documented default) rather than hard-coded.
- [ ] `cdk synth` / unit tests confirm the lifecycle rule is present with the expected transition + expiration.
- [ ] README / docs note the access-logs retention behavior and how to tune it.
## Additional notes
- `objectOwnership: OBJECT_WRITER` must be preserved (required for CloudFront logging).
- This change is backward compatible; existing objects are simply transitioned/expired according to the new rule.
Contributor guide
Research direction
Start in packages/cdk/lib/agentcore-stack.ts and inspect the AccessLogsBucket lifecycle rule and the existing env/stack configuration mechanism. Run cdk synth and the unit tests that cover the stack to verify the transition and configurable expiration. Done means the documented default, retention behavior, and tuning instructions are present in the README or docs while object ownership remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100