aws / aws/aws-cli

GuardDuty CreatePublishingDestination/UpdatePublishingDestination fails with misleading "resource folder does not exist" when the S3 folder-marker object has been expired by a bucket lifecycle rule

Open
#10,483 2 comments 0 reactions 1 assignee Claimed by @RyanFitzSimmonsAK View on GitHub
bug guardduty p2
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the bug

When `destinationArn` includes a prefix (e.g. `arn:aws:s3:::my-bucket/guardduty`), GuardDuty validates that the prefix exists as a folder — specifically, an S3 object with the key `guardduty/`. This validation is not documented as a hard requirement.

This silently breaks any Infrastructure-as-Code / CI-driven configuration flow (Terraform, CloudFormation custom resources, ad-hoc scripts) that re-asserts the publishing destination on every deploy against a bucket with lifecycle rules. The failure is invisible in day-to-day operations because findings keep flowing; it only surfaces on the next configuration change, often long after the underlying state was silently invalidated.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

- UpdatePublishingDestination with unchanged properties should succeed as a no-op. The destination is already in PUBLISHING state and continuously writing to the same destinationArn. Re-validating folder-marker existence adds no safety and breaks idempotent configuration flows (Terraform, CloudFormation custom resources, ad-hoc scripts).
- CreatePublishingDestination against a same-account bucket should not depend on a pre-existing folder marker. The documentation already commits to GuardDuty creating the "location" for same-account setups; that same auto-creation should tolerate — or restore — a marker that was subsequently deleted (e.g., by a bucket lifecycle rule) rather than failing the API call.
- If the folder-marker requirement is intentional and non-negotiable, then at minimum:
- The error message should identify what's actually being validated (e.g. "The S3 object '/' does not exist") rather than the ambiguous "resource folder does not exist," which sends operators down S3-permissions and bucket-existence rabbit holes.
- The requirement should be explicitly documented on the Exporting findings to S3 page, including its interaction with bucket lifecycle rules — since expiring lifecycle rules on log/audit buckets are common and this interaction is silent and non-obvious.

### Current Behavior

The initial `CreatePublishingDestination` call succeeds and (per the docs) GuardDuty creates the folder marker automatically. However, if the destination bucket has any S3 lifecycle rule that expires objects (a very common configuration for log/audit buckets), the folder marker is deleted along with everything else. Findings continue to be published successfully — they're written as full-path objects under the prefix, independent of the marker — so the destination remains in `PUBLISHING` state indefinitely.

But **any subsequent** `CreatePublishingDestination` or `UpdatePublishingDestination` call — including a no-op update with unchanged properties — fails with:

```
BadRequestException: The request failed because the resource folder specified in the destinationArn parameter does not exist.
Type: InvalidInputException
```

Workaround: `PutObject` a zero-byte folder marker (`aws s3api put-object --bucket --key /`) immediately before every `Create`/`Update` call.

### Reproduction Steps

1. Create an S3 bucket with a lifecycle rule that expires objects (e.g. 30 days, no filter).
2. Call `CreatePublishingDestination` with a `destinationArn` that includes a prefix. Verify success and that GuardDuty is `PUBLISHING`.
3. Wait for the lifecycle rule to expire the folder marker (`aws s3api head-object --key /` returns 404).
4. Call `CreatePublishingDestination` or `UpdatePublishingDestination` again with any properties. Fails with the above error.

### Possible Solution

Either:
- The API should not depend on the folder marker's continued existence (findings clearly don't), and the underlying validation should tolerate its absence and/or recreate it, **or**
- The error message should accurately describe what is being validated (e.g. "the S3 object `/` was not found") so operators can diagnose it without inspecting internals.

### Additional Information/Context

_No response_

### CLI version used

2.35.15

### Environment details (OS name and version, etc.)

Ubuntu 24.04.4 LTS (Github Runner)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.