ailia-ai / ailia-ai/ailia-sdk-cpp

ailiaOpenStreamFileW / ailiaUpdate: Infinite loop (CPU hang) on malformed Caffe prototxt — two separate DoS vectors

Abierto
#8 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
1
Forks
1
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

## Summary

Two denial-of-service vulnerabilities were found in `ailia.dll` that cause the calling thread to spin indefinitely when processing a malformed Caffe-format `.prototxt` file via the public API. Both issues were confirmed live against the Windows x64 release of the ailia SDK.

---

### Issue 1 — Parser infinite loop in `ailiaOpenStreamFileW`

**Affected API:** `ailiaOpenStreamFileW` (and `ailiaOpenStreamEx`)

**Root cause:**
Inside the Caffe prototxt text parser, the stream-state check after reading a field value uses the mask `(state & 6)` — which tests only `failbit (0x2)` and `badbit (0x4)`. When the file ends mid-value (e.g. `bottom:` with no value before EOF), `eofbit (0x1)` is set but the outer loop condition `(state & 6) == 0` remains true, so the loop never exits.

**Minimal reproducer (18 bytes):**

```
name:"tiny"
layer {
bottom:
```

Pass this as the prototxt path to `ailiaOpenStreamFileW`. The calling thread immediately enters a busy-loop consuming 100% of one CPU core and never returns.

**Call stack (observed, ailia.dll loaded at `0x00007FFE29EE0000`):**

```
ailia!ailiaFormatConvert+0x2aec01 ← infinite loop body
ailia!ailiaFormatConvert+0x2ade75
ailia!ailiaOpenStreamEx+0x726
ailia!ailiaOpenStreamFileW+0x145 ← public API entry point
```

**Decompiled loop condition (Ghidra, `FUN_1802bf670`):**

```c
while (uVar4 != 0xffffffff) { // snextc() loops until EOF
/* append char to buffer */
uVar4 = snextc(streambuf);
}
uVar4 = 1; // set eofbit only
if (bVar2) goto skip_failbit; // char was read → skip setting failbit!
uVar4 |= 2; // failbit — never reached when bVar2=true
setstate(uVar4);
// outer loop checks (state & 6) == 0 → true even with eofbit set → loops forever
```

---

### Issue 2 — Integer overflow in `ailiaUpdate` causes infinite loop

**Affected API:** `ailiaUpdate`

**Root cause:**
The convolution output-size formula uses a 32-bit signed intermediate. For a Caffe prototxt with `pad_h` / `pad_w` set to values near `(INT32_MAX − 10) / 2`, the expression `2 × pad_h + 10` overflows signed int32, producing a negative `output_h`. A downstream loop then spins on the negative size indefinitely.

**Minimal reproducer:**
Take any valid Caffe prototxt that ailia accepts, replace every `pad_h` and `pad_w` value with `1073741820`, load it with `ailiaOpenStreamFileW` + `ailiaOpenWeightFileW` (any matching caffemodel), then call `ailiaUpdate`. The function never returns.

```
pad_h: 1073741820
pad_w: 1073741820
```

**Arithmetic:**

```
output_h = (input_h + 2 × pad_h − kernel_h) / stride_h + 1
= (16 + 2 × 1073741820 − 7) / 1 + 1
= 2147483650 ← overflows int32
= −2147483646 ← negative → infinite loop
```

**Boundary observations** (kernel_h=7, input=16×16):

| pad_h | Behaviour |
|-----------------|------------------------------------------------------|
| ≤ 896 | `ailiaUpdate` returns 0 (success) |
| 960 – 65534 | Returns −5 or −128 — graceful error |
| **1073741820** | **`ailiaUpdate` never returns — CPU hang** |
| 2147483648 | Returns 0 with degenerate 0×0 output (harmless) |

---

### Environment

| Item | Value |
|---------------|----------------------------------------------------|
| OS | Windows 11 Pro 10.0.26200 (x64) |
| ailia.dll | version as shipped with CLIP STUDIO PAINT 1.5 |
| Architecture | x64 |
| API language | C / Python ctypes |

---

### Impact

Both bugs allow a crafted `.prototxt` file to cause 100% CPU utilisation on one core with no timeout or error recovery in the calling process. Any application that passes user-controlled or externally-sourced prototxt files to `ailiaOpenStreamFileW` / `ailiaUpdate` is affected.

**Suggested fix:**
- Issue 1: After reading a field value, check `eofbit` in addition to `failbit`/`badbit`, or call `good()` / `fail()` to test the full stream state.
- Issue 2: Widen the `output_h` intermediate to `int64_t` (or `size_t`) before computing the size, and add a negative-result guard before allocation.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.