apache / apache/pulsar

[Bug] enable debug log lead to pulsar offload to S3 service failed, because BlobStoreManagedLedgerOffloader#offload close BlockAwareSegmentInputStream third times and throw exception.

Open
#18,536 2 comments 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

2.10.1

### Minimal reproduce step

S3 service use huawei cloud obs service,
BlobStoreManagedLedgerOffloader#offload line: 214
when trigger offload, blockStream will close both writeBlobStore.uploadMultipartPart(mpu, partId, partPayload) and try (BlockAwareSegmentInputStream blockStream = new BlockAwareSegmentInputStreamImpl(
readHandle, startEntry, blockSize)) will close blockStream, when finish try blockStream.close will throw exception
Illegal ReferenceCountException: refCnt: 0, decrement: 1
code:
try {
long startEntry = 0;
int partId = 1;
long entryBytesWritten = 0;
while (startEntry <= readHandle.getLastAddConfirmed()) {
int blockSize = BlockAwareSegmentInputStreamImpl
.calculateBlockSize(config.getMaxBlockSizeInBytes(), readHandle, startEntry, entryBytesWritten);

try (BlockAwareSegmentInputStream blockStream = new BlockAwareSegmentInputStreamImpl(
readHandle, startEntry, blockSize)) {

Payload partPayload = Payloads.newInputStreamPayload(blockStream);
partPayload.getContentMetadata().setContentLength((long) blockSize);
partPayload.getContentMetadata().setContentType("application/octet-stream");
parts.add(writeBlobStore.uploadMultipartPart(mpu, partId, partPayload));
log.debug("UploadMultipartPart. container: {}, blobName: {}, partId: {}, mpu: {}",
config.getBucket(), dataBlockKey, partId, mpu.id());

indexBuilder.addBlock(startEntry, partId, blockSize);

if (blockStream.getEndEntryId() != -1) {
startEntry = blockStream.getEndEntryId() + 1;
} else {
// could not read entry from ledger.
break;
}
entryBytesWritten += blockStream.getBlockEntryBytesCount();
partId++;
}

dataObjectLength += blockSize;
}

writeBlobStore.completeMultipartUpload(mpu, parts);
mpu = null;
} catch (Throwable t) {
try {
if (mpu != null) {
writeBlobStore.abortMultipartUpload(mpu);
}
} catch (Throwable throwable) {
log.error("Failed abortMultipartUpload in bucket - {} with key - {}, uploadId - {}.",
config.getBucket(), dataBlockKey, mpu.id(), throwable);
}
promise.completeExceptionally(t);
return;
}
![image](https://user-images.githubusercontent.com/54945799/202666728-7d2ed7a7-4099-438a-9d74-12b98b40262c.png)

### What did you expect to see?

no exception occurs, ledger log send to obs file success, but actually when send part file sucess once, code throw exception lead to delete uploaded file in catch exception code.

### What did you see instead?

no exception occurs, ledger log send to obs file success, but actually when send part file sucess once, code throw exception lead to delete uploaded file in catch exception code.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start at BlobStoreManagedLedgerOffloader#offload around line 214 and inspect how uploadMultipartPart and the try-with-resources block close BlockAwareSegmentInputStream. Reproduce the offload against Huawei Cloud OBS and verify that a successful part upload does not trigger Illegal ReferenceCountException or the abort path, while the completed ledger log remains in OBS.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.