apache / apache/cloudstack

Netty IllegalReferenceCountException causes false upload failure despite successful upload

オープン
#12,489 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
component:secondary-storage component:ssvm Severity:Minor type:bug
主要言語
Java
スター
3.1k
フォーク
1.4k
平均マージ
6日 19時間
マージ済み PR(30日)
32

説明

### problem

**Related Issue**
Related to https://github.com/apache/cloudstack/issues/12486

**Summary**
When uploading templates/ISOs from local file via the UI, the upload completes successfully on the SSVM (file is written to disk, SSVM returns HTTP 200 "upload successful"), but a Netty IllegalReferenceCountException occurs during response handling. This causes the UI to display an error to the user even though the upload actually succeeded. The template eventually becomes "Ready" after background processing completes.

### versions

CloudStack Version 4.20.2.0

### The steps to reproduce the bug

1. Navigate to Images → Templates
2. Click "Upload Template from local"
4. Fill in the form and submit
5. Watch browser Network tab - upload returns 200 OK
6. Check browser Console - may show error despite 200 OK
7. Check SSVM logs - see Netty IllegalReferenceCountException
8. Wait - template eventually becomes Ready

Note: This issue is intermittent.

**Expected Behavior**
When the upload succeeds (HTTP 200, file written to disk), the UI should show success to the user.

**Actual Behavior**

- Upload succeeds (HTTP 200, "upload successful")
- File is written to disk correctly
- Netty throws IllegalReferenceCountException during cleanup
- UI may show error to user (HTTP 432 in some cases)
- Template eventually becomes Ready after background processing

**Impact**

- User confusion: User sees error but upload actually worked
- Duplicate uploads: User may retry, creating duplicate templates
- Wasted time: User troubleshoots non-existent problem

**Root Cause Analysis**
The IllegalReferenceCountException occurs in `HttpUploadServerHandler.channelUnregistered()` when trying to destroy the `HttpPostRequestDecoder`. The decoder's reference count has already reached 0, indicating it was already released elsewhere.
This is a reference counting bug in the Netty upload handler - the decoder is being released twice:

- Once during normal processing
- Again in `channelUnregistered()` cleanup

**Evidence**

Browser Network Tab - Upload Succeeded (HTTP 200)
Request Headers (all correct):

```
X-Expires: 2026-01-21T16:46:06.391Z
X-Metadata: 7d9tasmSFbD/i/a+7cIPVHUKLB5oZUTfHOf6nIwEo6gdgJ1IHcgXfSHe1TouKFGRARpVRJJyyfVX5fqGcsv14oYurocRB4KGOkuW6zfW3GZC3hxhM2vA5OjK5Q497cxlc/TkgD/BOYoyx3VS32w4jFFxfVty9YSrc2tx1/bo9kE4MFXbFiuGNHI799xK8WhEXiTuR/7iyxRqLM17+ql/JYxlPLSP7DWbVOMFwm7HACbmSSaUNyJhi2OiaOmmU0TC5jpezvLxXhVmDsl9Fe9/uhTG2PDtmd/d2L+QtQz8+zNqxwgrS8EJqJKsk7q7tPzoDFKWc7Qp0uYpPvMqpyd377kXA2qhKnphEKrB6rBvIHV0OaX071UMNSysfrYC+GOIi3smUK6p5x939GlNlVFI6kYM5pkVZ9AsuNzGii4SMDZc8UGYrYmhazT10+ye0IlAVw5cFu5hcwVmeGnK8O2xDQkHo1z5wFYxiqCzZSuVdrhk0jdV0ktZ0CAcQeZzjBblu4DWFCcdqRDHl+8SkeAwtUjV3ZDht0u4D1lQdgV4G/NeDPCPhkm61WhKcR4AHvq5pb1+bs9h/8+b2BkuhfNbdNCd/gjW2kMsl4KcbhQwB+3uQsxZ9srg+hKacPB2Cyd8kgdLzuC4zYBq8mv8pRg4gpj5fbVC5Vs7pYbzpDlfoeIAuCZS7pDJM4Jontlae/QoGGo5
X-Signature: zw+vnaAEiYNsjUfHov44KNZ/GBw=
Content-Length: 400621794
```

- Response:

```
Status Code: 200 OK
Body: upload successful.
```

- SSVM Logs (/var/log/cloud.log) - Upload Received and Processed

```
2026-01-21T15:46:06,551 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: SIGNATURE=zw+vnaAEiYNsjUfHov44KNZ/GBw=
2026-01-21T15:46:06,552 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: METADATA=7d9tasmSFbD/i/a+7cIPVHUKLB5oZUT...
2026-01-21T15:46:06,552 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: EXPIRES=2026-01-21T16:46:06.391Z
2026-01-21T15:46:06,552 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: HOST=10.0.56.181
2026-01-21T15:46:06,553 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: CONTENT_LENGTH=400621794
2026-01-21T15:46:06,554 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) URI: uuid=6c80606a-8826-460f-88a8-c15ed3606a6c
2026-01-21T15:46:06,555 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) base directory: /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203
2026-01-21T15:46:06,611 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) BODY: FileUpload: name="file"; filename="linux-debian-11-x86_64-gen2-v1.qcow2"
```

- SSVM Logs - Netty Exception During Cleanup

```
2026-01-21T15:47:30,572 WARN [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[])
Decoder already destroyed
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:140)
at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:79)
at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:293)
at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelUnregistered(HttpUploadServerHandler.java:169)
```

- File Successfully Written to SSVM

```
root@s-1-VM:~# ls -la /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203/
total 391236
drwxrwxrwx 2 root root 95 Jan 21 2026 .
drwxr-xr-x 3 root root 25 Jan 21 2026 ..
-rw-rw-rw- 1 root root 400621568 Jan 21 2026 c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2
-rw-rw-rw- 1 root root 340 Jan 21 2026 template.properties
```

- Template Properties File Created

```
root@s-1-VM:~# cat /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203/template.properties
#
#Wed Jan 21 15:47:30 UTC 2026
uniquename=203-2-0cb72ddc-d73a-3f2c-9f1f-24af1d033391
qcow2.filename=c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2
qcow2.virtualsize=5368709120
description=test
qcow2.size=400621568
filename=c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2
size=400621568
public=true
id=203
hvm=true
virtualsize=5368709120
qcow2=true
```

- Despite the error shown to the user, the template eventually became Ready:

```
{
"id": "6c80606a-8826-460f-88a8-c15ed3606a6c",
"name": "Debian 11",
"isready": true,
"status": "Download Complete",
"downloaddetails": [
{
"datastore": "NFS://10.0.32.4/acs/secondary/ref-trl-10677-k-Mr8-rositsa-kyuchukova/ref-trl-10677-k-Mr8-rositsa-kyuchukova-sec1",
"downloadPercent": "100",
"downloadState": "DOWNLOADED"
}
],
"physicalsize": 400621568,
"size": 5368709120
}
```

### What to do about it?

In H`ttpUploadServerHandler.java`, add a null check or try-catch around the decoder destruction in `channelUnregistered()`

**Workaround**

Ignore the error in the UI and wait - template will eventually become Ready
Refresh the template list after a minute to see actual status

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

H`ttpUploadServerHandler.java`、特に channelUnregistered() を読み、通常のアップロード処理中に HttpPostRequestDecoder が解放される箇所を追跡してください。ローカルファイルのアップロードを再現し、HTTP 200 応答、書き込まれたファイル、テンプレートの完了が、クリーンアップ例外や誤った UI エラーなしに引き続き成功することを確認してください。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
backend
issue の種類
バグ
難易度
2/5
見積もり時間
1〜3時間
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
45/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。