ailia-ai / ailia-ai/ailia-sdk-cpp

ailiaOpenStreamFileW / ailiaUpdate: Infinite loop (CPU hang) on malformed Caffe prototxt — two separate DoS vectors

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Summary

Two denial-of-service vulnerabilities were found in `ailia.dll` that cause the calling thread to spin indefinitely when processing a malformed Caffe-format `.prototxt` file via the public API. Both issues were confirmed live against the Windows x64 release of the ailia SDK.

---

### Issue 1 — Parser infinite loop in `ailiaOpenStreamFileW`

**Affected API:** `ailiaOpenStreamFileW` (and `ailiaOpenStreamEx`)

**Root cause:**
Inside the Caffe prototxt text parser, the stream-state check after reading a field value uses the mask `(state & 6)` — which tests only `failbit (0x2)` and `badbit (0x4)`. When the file ends mid-value (e.g. `bottom:` with no value before EOF), `eofbit (0x1)` is set but the outer loop condition `(state & 6) == 0` remains true, so the loop never exits.

**Minimal reproducer (18 bytes):**

```
name:"tiny"
layer {
bottom:
```

Pass this as the prototxt path to `ailiaOpenStreamFileW`. The calling thread immediately enters a busy-loop consuming 100% of one CPU core and never returns.

**Call stack (observed, ailia.dll loaded at `0x00007FFE29EE0000`):**

```
ailia!ailiaFormatConvert+0x2aec01 ← infinite loop body
ailia!ailiaFormatConvert+0x2ade75
ailia!ailiaOpenStreamEx+0x726
ailia!ailiaOpenStreamFileW+0x145 ← public API entry point
```

**Decompiled loop condition (Ghidra, `FUN_1802bf670`):**

```c
while (uVar4 != 0xffffffff) { // snextc() loops until EOF
/* append char to buffer */
uVar4 = snextc(streambuf);
}
uVar4 = 1; // set eofbit only
if (bVar2) goto skip_failbit; // char was read → skip setting failbit!
uVar4 |= 2; // failbit — never reached when bVar2=true
setstate(uVar4);
// outer loop checks (state & 6) == 0 → true even with eofbit set → loops forever
```

---

### Issue 2 — Integer overflow in `ailiaUpdate` causes infinite loop

**Affected API:** `ailiaUpdate`

**Root cause:**
The convolution output-size formula uses a 32-bit signed intermediate. For a Caffe prototxt with `pad_h` / `pad_w` set to values near `(INT32_MAX − 10) / 2`, the expression `2 × pad_h + 10` overflows signed int32, producing a negative `output_h`. A downstream loop then spins on the negative size indefinitely.

**Minimal reproducer:**
Take any valid Caffe prototxt that ailia accepts, replace every `pad_h` and `pad_w` value with `1073741820`, load it with `ailiaOpenStreamFileW` + `ailiaOpenWeightFileW` (any matching caffemodel), then call `ailiaUpdate`. The function never returns.

```
pad_h: 1073741820
pad_w: 1073741820
```

**Arithmetic:**

```
output_h = (input_h + 2 × pad_h − kernel_h) / stride_h + 1
= (16 + 2 × 1073741820 − 7) / 1 + 1
= 2147483650 ← overflows int32
= −2147483646 ← negative → infinite loop
```

**Boundary observations** (kernel_h=7, input=16×16):

| pad_h | Behaviour |
|-----------------|------------------------------------------------------|
| ≤ 896 | `ailiaUpdate` returns 0 (success) |
| 960 – 65534 | Returns −5 or −128 — graceful error |
| **1073741820** | **`ailiaUpdate` never returns — CPU hang** |
| 2147483648 | Returns 0 with degenerate 0×0 output (harmless) |

---

### Environment

| Item | Value |
|---------------|----------------------------------------------------|
| OS | Windows 11 Pro 10.0.26200 (x64) |
| ailia.dll | version as shipped with CLIP STUDIO PAINT 1.5 |
| Architecture | x64 |
| API language | C / Python ctypes |

---

### Impact

Both bugs allow a crafted `.prototxt` file to cause 100% CPU utilisation on one core with no timeout or error recovery in the calling process. Any application that passes user-controlled or externally-sourced prototxt files to `ailiaOpenStreamFileW` / `ailiaUpdate` is affected.

**Suggested fix:**
- Issue 1: After reading a field value, check `eofbit` in addition to `failbit`/`badbit`, or call `good()` / `fail()` to test the full stream state.
- Issue 2: Widen the `output_h` intermediate to `int64_t` (or `size_t`) before computing the size, and add a negative-result guard before allocation.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.