apache / apache/pulsar-helm-chart

Ledgers not being offloaded to AWS S3 during testing

Open
#623 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
240
Forks
260
Avg merge
4h 12m
Merged PRs (30d)
1

Description

**Describe the bug**
Using 4.1.0 version of this helm chart.
Everything appears correct on the broker's env:

```
s3ManagedLedgerOffloadServiceEndpoint=https://s3.eu-west-1.amazonaws.com
AWS_DEFAULT_REGION=eu-west-1
AWS_REGION=eu-west-1
AWS_ROLE_ARN=arn:aws:iam::111111111111:role/eks-pulsar
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
managedLedgerOffloadDriver=aws-s3
AWS_STS_REGIONAL_ENDPOINTS=regional
s3ManagedLedgerOffloadRegion=eu-west-1
offloadersDirectory=/pulsar/offloaders
PULSAR_PREFIX_managedLedgerOffloadThresholdInBytes=10
loggerLevelUpdates=org.apache.bookkeeper.mledger.offload=DEBUG,org.jclouds=DEBUG
PULSAR_PREFIX_managedLedgerOffloadDeletionLagInMillis=0
managedLedgerOffloadDriver=aws-s3
s3ManagedLedgerOffloadBucket=mybucketishere
```

**To Reproduce**
Steps to reproduce the behavior:
Use these broker settings (warning: settings are set to offload fast for testing)

```
broker:
podSecurityContext: # << testing may not be needed. please advise
enabled: true
fsGroup: 10000
service_account:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/eks-pulsar
storageOffload:
driver: aws-s3
bucket: "had-no-choice-but-to-hardcode-no-extraenvs-worked-in-these-keys"
region: "had-no-choice-but-to-hardcode-no-extraenvs-worked-in-these-key"
managedLedgerOffloadAutoTriggerSizeThresholdBytes: "10"
managedLedgerOffloadDeletionLagMs: "0"
configData:
acknowledgmentAtBatchIndexLevelEnabled: "true"
brokerDeleteInactiveTopicsEnabled: "false"
offloadersDirectory: "/pulsar/offloaders". # << may not be needed
managedLedgerMinLedgerRolloverTimeMinutes: "5"
managedLedgerMaxLedgerSize: "256M"
s3ManagedLedgerOffloadServiceEndpoint: "https://s3.eu-west-1.amazonaws.com"
loggerLevelUpdates: "org.apache.bookkeeper.mledger.offload=DEBUG,org.jclouds=DEBUG"
```

Also, extraenvvars for broker did not work for me for anything in the broker key, but I may create another ticket for that as it's much less blocking.

**Expected behavior**
It should offload within 5 minutes when a ledger is closed to my knowledge. I tried running commands to roll it over manually and it never rolled over. I tried consuming and it didn't make a difference.

**Logs**

```
Offload triggered for persistent://public/default/offload-test-3 for messages before 793:0:-1

2025-08-01T04:24:39,760+0000 [pulsar-web-40-5] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/offload-test-3] Starting offload operation at messageId 793:0:-1

2025-08-01T04:24:39,761+0000 [pulsar-web-40-5] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/offload-test-3] Start ledgersOffload. ledgers=[793] totalSize=842208

2025-08-01T04:24:39,761+0000 [pulsar-web-40-5] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/offload-test-3] No ledgers to offload

2025-08-01T04:24:39,761+0000 [pulsar-web-40-5] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/offload-test-3] Completed successfully offload operation at messageId 793:0:-1

2025-08-01T04:24:39,762+0000 [pulsar-web-40-5] INFO  org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [01/Aug/2025:04:24:39 +0000] "PUT /admin/v2/persistent/public/default/offload-test-3/offload HTTP/1.1" 204 0 "-" "Pulsar-Java-v4.0.5" 4
```

**Desktop (please complete the following information):**
- OS: [e.g. iOS]

**Additional context**
Would appreciate any help here. I can also definitely contribute to the env vars. IMHO extraenvvars for the entire chart would make it very malleable and repeatable for people with many clusters.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.