apache / apache/cloudstack

Netty IllegalReferenceCountException causes false upload failure despite successful upload

Đang mở
#12,489 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
component:secondary-storage component:ssvm Severity:Minor type:bug
Ngôn ngữ chính
Java
Star
3.1k
Fork
1.4k
Merge trung bình
6 ngày 19 giờ
Pull request đã merge (30 ngày)
32

Mô tả

### problem

**Related Issue**
Related to https://github.com/apache/cloudstack/issues/12486

**Summary**
When uploading templates/ISOs from local file via the UI, the upload completes successfully on the SSVM (file is written to disk, SSVM returns HTTP 200 "upload successful"), but a Netty IllegalReferenceCountException occurs during response handling. This causes the UI to display an error to the user even though the upload actually succeeded. The template eventually becomes "Ready" after background processing completes.

### versions

CloudStack Version 4.20.2.0

### The steps to reproduce the bug

1. Navigate to Images → Templates
2. Click "Upload Template from local"
4. Fill in the form and submit
5. Watch browser Network tab - upload returns 200 OK
6. Check browser Console - may show error despite 200 OK
7. Check SSVM logs - see Netty IllegalReferenceCountException
8. Wait - template eventually becomes Ready

Note: This issue is intermittent.

**Expected Behavior**
When the upload succeeds (HTTP 200, file written to disk), the UI should show success to the user.

**Actual Behavior**

- Upload succeeds (HTTP 200, "upload successful")
- File is written to disk correctly
- Netty throws IllegalReferenceCountException during cleanup
- UI may show error to user (HTTP 432 in some cases)
- Template eventually becomes Ready after background processing

**Impact**

- User confusion: User sees error but upload actually worked
- Duplicate uploads: User may retry, creating duplicate templates
- Wasted time: User troubleshoots non-existent problem

**Root Cause Analysis**
The IllegalReferenceCountException occurs in `HttpUploadServerHandler.channelUnregistered()` when trying to destroy the `HttpPostRequestDecoder`. The decoder's reference count has already reached 0, indicating it was already released elsewhere.
This is a reference counting bug in the Netty upload handler - the decoder is being released twice:

- Once during normal processing
- Again in `channelUnregistered()` cleanup

**Evidence**

Browser Network Tab - Upload Succeeded (HTTP 200)
Request Headers (all correct):

```
X-Expires: 2026-01-21T16:46:06.391Z
X-Metadata: 7d9tasmSFbD/i/a+7cIPVHUKLB5oZUTfHOf6nIwEo6gdgJ1IHcgXfSHe1TouKFGRARpVRJJyyfVX5fqGcsv14oYurocRB4KGOkuW6zfW3GZC3hxhM2vA5OjK5Q497cxlc/TkgD/BOYoyx3VS32w4jFFxfVty9YSrc2tx1/bo9kE4MFXbFiuGNHI799xK8WhEXiTuR/7iyxRqLM17+ql/JYxlPLSP7DWbVOMFwm7HACbmSSaUNyJhi2OiaOmmU0TC5jpezvLxXhVmDsl9Fe9/uhTG2PDtmd/d2L+QtQz8+zNqxwgrS8EJqJKsk7q7tPzoDFKWc7Qp0uYpPvMqpyd377kXA2qhKnphEKrB6rBvIHV0OaX071UMNSysfrYC+GOIi3smUK6p5x939GlNlVFI6kYM5pkVZ9AsuNzGii4SMDZc8UGYrYmhazT10+ye0IlAVw5cFu5hcwVmeGnK8O2xDQkHo1z5wFYxiqCzZSuVdrhk0jdV0ktZ0CAcQeZzjBblu4DWFCcdqRDHl+8SkeAwtUjV3ZDht0u4D1lQdgV4G/NeDPCPhkm61WhKcR4AHvq5pb1+bs9h/8+b2BkuhfNbdNCd/gjW2kMsl4KcbhQwB+3uQsxZ9srg+hKacPB2Cyd8kgdLzuC4zYBq8mv8pRg4gpj5fbVC5Vs7pYbzpDlfoeIAuCZS7pDJM4Jontlae/QoGGo5
X-Signature: zw+vnaAEiYNsjUfHov44KNZ/GBw=
Content-Length: 400621794
```

- Response:

```
Status Code: 200 OK
Body: upload successful.
```

- SSVM Logs (/var/log/cloud.log) - Upload Received and Processed

```
2026-01-21T15:46:06,551 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: SIGNATURE=zw+vnaAEiYNsjUfHov44KNZ/GBw=
2026-01-21T15:46:06,552 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: METADATA=7d9tasmSFbD/i/a+7cIPVHUKLB5oZUT...
2026-01-21T15:46:06,552 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: EXPIRES=2026-01-21T16:46:06.391Z
2026-01-21T15:46:06,552 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: HOST=10.0.56.181
2026-01-21T15:46:06,553 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: CONTENT_LENGTH=400621794
2026-01-21T15:46:06,554 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) URI: uuid=6c80606a-8826-460f-88a8-c15ed3606a6c
2026-01-21T15:46:06,555 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) base directory: /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203
2026-01-21T15:46:06,611 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) BODY: FileUpload: name="file"; filename="linux-debian-11-x86_64-gen2-v1.qcow2"
```

- SSVM Logs - Netty Exception During Cleanup

```
2026-01-21T15:47:30,572 WARN [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[])
Decoder already destroyed
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:140)
at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:79)
at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:293)
at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelUnregistered(HttpUploadServerHandler.java:169)
```

- File Successfully Written to SSVM

```
root@s-1-VM:~# ls -la /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203/
total 391236
drwxrwxrwx 2 root root 95 Jan 21 2026 .
drwxr-xr-x 3 root root 25 Jan 21 2026 ..
-rw-rw-rw- 1 root root 400621568 Jan 21 2026 c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2
-rw-rw-rw- 1 root root 340 Jan 21 2026 template.properties
```

- Template Properties File Created

```
root@s-1-VM:~# cat /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203/template.properties
#
#Wed Jan 21 15:47:30 UTC 2026
uniquename=203-2-0cb72ddc-d73a-3f2c-9f1f-24af1d033391
qcow2.filename=c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2
qcow2.virtualsize=5368709120
description=test
qcow2.size=400621568
filename=c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2
size=400621568
public=true
id=203
hvm=true
virtualsize=5368709120
qcow2=true
```

- Despite the error shown to the user, the template eventually became Ready:

```
{
"id": "6c80606a-8826-460f-88a8-c15ed3606a6c",
"name": "Debian 11",
"isready": true,
"status": "Download Complete",
"downloaddetails": [
{
"datastore": "NFS://10.0.32.4/acs/secondary/ref-trl-10677-k-Mr8-rositsa-kyuchukova/ref-trl-10677-k-Mr8-rositsa-kyuchukova-sec1",
"downloadPercent": "100",
"downloadState": "DOWNLOADED"
}
],
"physicalsize": 400621568,
"size": 5368709120
}
```

### What to do about it?

In H`ttpUploadServerHandler.java`, add a null check or try-catch around the decoder destruction in `channelUnregistered()`

**Workaround**

Ignore the error in the UI and wait - template will eventually become Ready
Refresh the template list after a minute to see actual status

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Đọc H`ttpUploadServerHandler.java`, đặc biệt là channelUnregistered(), và theo dõi nơi HttpPostRequestDecoder được giải phóng trong quá trình xử lý upload bình thường. Tái hiện việc upload một tệp cục bộ và xác nhận rằng phản hồi HTTP 200, tệp đã được ghi và việc hoàn tất template vẫn thành công mà không có ngoại lệ cleanup hoặc lỗi UI giả.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
java
Lĩnh vực
backend
Loại issue
Lỗi
Độ khó
2/5
Thời gian dự kiến
1-3 giờ
Mức độ hoạt động
Đình trệ
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
45/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.