apache / apache/pulsar-helm-chart
Ledgers not being offloaded to AWS S3 during testing
- Dominant language
- Shell
- Stars
- 240
- Forks
- 260
- Avg merge
- 4h 12m
- Merged PRs (30d)
- 1
Description
**Describe the bug**
Using 4.1.0 version of this helm chart.
Everything appears correct on the broker's env:
```
s3ManagedLedgerOffloadServiceEndpoint=https://s3.eu-west-1.amazonaws.com
AWS_DEFAULT_REGION=eu-west-1
AWS_REGION=eu-west-1
AWS_ROLE_ARN=arn:aws:iam::111111111111:role/eks-pulsar
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
managedLedgerOffloadDriver=aws-s3
AWS_STS_REGIONAL_ENDPOINTS=regional
s3ManagedLedgerOffloadRegion=eu-west-1
offloadersDirectory=/pulsar/offloaders
PULSAR_PREFIX_managedLedgerOffloadThresholdInBytes=10
loggerLevelUpdates=org.apache.bookkeeper.mledger.offload=DEBUG,org.jclouds=DEBUG
PULSAR_PREFIX_managedLedgerOffloadDeletionLagInMillis=0
managedLedgerOffloadDriver=aws-s3
s3ManagedLedgerOffloadBucket=mybucketishere
```
**To Reproduce**
Steps to reproduce the behavior:
Use these broker settings (warning: settings are set to offload fast for testing)
```
broker:
podSecurityContext: # << testing may not be needed. please advise
enabled: true
fsGroup: 10000
service_account:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/eks-pulsar
storageOffload:
driver: aws-s3
bucket: "had-no-choice-but-to-hardcode-no-extraenvs-worked-in-these-keys"
region: "had-no-choice-but-to-hardcode-no-extraenvs-worked-in-these-key"
managedLedgerOffloadAutoTriggerSizeThresholdBytes: "10"
managedLedgerOffloadDeletionLagMs: "0"
configData:
acknowledgmentAtBatchIndexLevelEnabled: "true"
brokerDeleteInactiveTopicsEnabled: "false"
offloadersDirectory: "/pulsar/offloaders". # << may not be needed
managedLedgerMinLedgerRolloverTimeMinutes: "5"
managedLedgerMaxLedgerSize: "256M"
s3ManagedLedgerOffloadServiceEndpoint: "https://s3.eu-west-1.amazonaws.com"
loggerLevelUpdates: "org.apache.bookkeeper.mledger.offload=DEBUG,org.jclouds=DEBUG"
```
Also, extraenvvars for broker did not work for me for anything in the broker key, but I may create another ticket for that as it's much less blocking.
**Expected behavior**
It should offload within 5 minutes when a ledger is closed to my knowledge. I tried running commands to roll it over manually and it never rolled over. I tried consuming and it didn't make a difference.
**Logs**
```
Offload triggered for persistent://public/default/offload-test-3 for messages before 793:0:-1
2025-08-01T04:24:39,760+0000 [pulsar-web-40-5] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/offload-test-3] Starting offload operation at messageId 793:0:-1
2025-08-01T04:24:39,761+0000 [pulsar-web-40-5] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/offload-test-3] Start ledgersOffload. ledgers=[793] totalSize=842208
2025-08-01T04:24:39,761+0000 [pulsar-web-40-5] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/offload-test-3] No ledgers to offload
2025-08-01T04:24:39,761+0000 [pulsar-web-40-5] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/offload-test-3] Completed successfully offload operation at messageId 793:0:-1
2025-08-01T04:24:39,762+0000 [pulsar-web-40-5] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [01/Aug/2025:04:24:39 +0000] "PUT /admin/v2/persistent/public/default/offload-test-3/offload HTTP/1.1" 204 0 "-" "Pulsar-Java-v4.0.5" 4
```
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
**Additional context**
Would appreciate any help here. I can also definitely contribute to the env vars. IMHO extraenvvars for the entire chart would make it very malleable and repeatable for people with many clusters.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.