Azure / Azure/Azurite

Table service corrupts multi-byte UTF-8 characters split across request body chunks (`readRequestIntoText` uses `segments.join` instead of `Buffer.concat`)

Open
#2,670 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
2.3k
Forks
393
Avg merge
1d 20h
Merged PRs (30d)
36

Description

### Which service(blob, file, queue, table) does this issue concern?

Table. (The same generated helper exists verbatim in the blob and queue serializers — see the last section; we verified the corruption for the table service.)

### Which version of the Azurite was used?

v3.35.0 (latest release). The affected code is unchanged on current `main` ([c9aa6ff](https://github.com/Azure/Azurite/commit/c9aa6fff44c6ec0e21f5f5aef8207790661719f3)).

### Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

npm. (The affected code ships identically in all distributions.)

### What's the Node.js version?

First hit on Node 20 (Linux CI); reproduced with Node 25 on macOS using the repro below. Not Node-version-specific.

### What problem was encountered?

When the request body of a table operation (e.g. an entity upsert) is delivered to Azurite's HTTP server in more than one `data` event and a multi-byte UTF-8 character straddles a chunk boundary, that character is decoded into U+FFFD replacement characters. The corrupted value is then **persisted** — every subsequent read returns it — and the string length changes too (a split 4-byte emoji becomes 4 replacement characters instead of 2 UTF-16 code units). Real Azure Table Storage handles the same request correctly.

Root cause — [`readRequestIntoText` in `src/table/generated/utils/serializer.ts`](https://github.com/Azure/Azurite/blob/c9aa6fff44c6ec0e21f5f5aef8207790661719f3/src/table/generated/utils/serializer.ts#L183-L196):

```ts
async function readRequestIntoText(req: IRequest): Promise {
return new Promise((resolve, reject) => {
const segments: string[] = [];
const bodyStream = req.getBodyStream();
bodyStream.on("data", buffer => {
segments.push(buffer);
});
bodyStream.on("error", reject);
bodyStream.on("end", () => {
const joined = segments.join(""); // <-- decodes every chunk in isolation
resolve(joined);
});
});
}
```

`getBodyStream()` returns the raw Express request with no encoding set ([`ExpressRequestAdapter.ts#L25-L27`](https://github.com/Azure/Azurite/blob/c9aa6fff44c6ec0e21f5f5aef8207790661719f3/src/table/generated/ExpressRequestAdapter.ts#L25-L27)), so the `data` events emit `Buffer`s (the `segments: string[]` annotation is misleading). `Array.prototype.join` then stringifies each buffer **individually**, and a UTF-8 sequence whose bytes are split across two chunks cannot be decoded by either side alone — both halves become U+FFFD.

Example: `🎉` is `F0 9F 8E 89`. A chunk boundary after `F0` yields `����` (1 + 3 replacement characters) instead of `🎉`.

Whether the bug fires depends only on how the OS delivers the body. Small loopback requests usually arrive in a single `data` event, which hides the bug; large bodies (a ~175 KiB entity-upsert JSON in our case) are reliably split into multiple events. For us the same test failed deterministically on Linux CI and passed on macOS loopback — Azurite silently diverges from the real service only under perfectly normal network segmentation, which made this hard to track down.

An Azurite `-d` debug log confirms the corruption happens while reading the request: the `deserialize(): Raw request body string is ...` line already contains the replacement characters.

### Steps to reproduce the issue?

Self-contained repro: `npm install @azure/data-tables`, start `azurite --inMemoryPersistence`, then `node repro.js`. A tiny TCP proxy re-segments the byte stream so the body reliably arrives in many chunks — it merely makes deterministic what a real network does with large bodies anyway:

```js
// repro.js — Azurite table service corrupts multi-byte UTF-8 split across body chunks
//
// npm install @azure/data-tables
// npx azurite --inMemoryPersistence --silent (table endpoint on 127.0.0.1:10002)
// node repro.js
//
// The tiny TCP proxy below re-segments the byte stream into 1031-byte slices with a
// small pause, so Azurite's HTTP server sees the request body as many 'data' events —
// it merely makes deterministic what a real network does with large bodies anyway.

const net = require("net");
const { TableClient, AzureNamedKeyCredential } = require("@azure/data-tables");

const AZURITE_PORT = 10002;
const PROXY_PORT = 10102;
const CHUNK = 1031; // odd on purpose: boundaries drift through every byte offset of the 4-byte sequences

const server = net.createServer((client) => {
const upstream = net.connect(AZURITE_PORT, "127.0.0.1");
client.on("data", async (data) => {
client.pause();
for (let i = 0; i < data.length; i += CHUNK) {
upstream.write(data.subarray(i, i + CHUNK));
await new Promise((r) => setTimeout(r, 2)); // each slice arrives as its own 'data' event
}
client.resume();
});
upstream.pipe(client);
client.on("error", () => upstream.destroy());
upstream.on("error", () => client.destroy());
});

server.listen(PROXY_PORT, async () => {
const client = new TableClient(
`http://127.0.0.1:${PROXY_PORT}/devstoreaccount1`,
"utf8repro",
new AzureNamedKeyCredential(
"devstoreaccount1",
"Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
),
{ allowInsecureConnection: true }
);
await client.createTable();

// 4 string properties, each below the 32K-character property limit,
// together ~240 KB of 4-byte UTF-8 on the wire
const value = "🎉".repeat(15000);
const props = ["p0", "p1", "p2", "p3"];
const entity = { partitionKey: "p", rowKey: "r" };
for (const k of props) entity[k] = value;
await client.upsertEntity(entity, "Replace");
const read = await client.getEntity("p", "r");

const intact = props.every((k) => read[k] === value);
const fffd = props.reduce((n, k) => n + ((read[k].match(/\uFFFD/g) || []).length), 0);
console.log("roundtrip intact :", intact);
console.log("chars/prop written:", value.length);
console.log("chars/prop read :", props.map((k) => read[k].length).join(", "));
console.log("U+FFFD read back :", fffd);
server.close();
process.exit(intact ? 0 : 1);
});
```

Output against unpatched Azurite v3.35.0 (reproduced on macOS and Linux; exact counts vary with segmentation):

```
roundtrip intact : false
chars/prop written: 30000
chars/prop read : 30045, 30043, 30044, 30045
U+FFFD read back : 529
```

Expected (= behavior of real Azure Table Storage):

```
roundtrip intact : true
chars/prop written: 30000
chars/prop read : 30000, 30000, 30000, 30000
U+FFFD read back : 0
```

### Have you found a mitigation/solution?

Yes — collect `Buffer`s and decode once over the concatenated bytes instead of per chunk:

```ts
const segments: Buffer[] = [];
...
bodyStream.on("end", () => {
resolve(Buffer.concat(segments).toString("utf8"));
});
```

(or stream-decode with `string_decoder.StringDecoder`). We verified that with exactly this one-line change applied to `dist/src/table/generated/utils/serializer.js`, the repro above round-trips losslessly (second output block). The repo already uses the correct pattern in [`TableBatchUtils.StreamToString`](https://github.com/Azure/Azurite/blob/c9aa6fff44c6ec0e21f5f5aef8207790661719f3/src/table/batch/TableBatchUtils.ts#L10-L22), so the table **batch** path is not affected.

Note: the identical `readRequestIntoText` also exists in the generated [blob](https://github.com/Azure/Azurite/blob/c9aa6fff44c6ec0e21f5f5aef8207790661719f3/src/blob/generated/utils/serializer.ts#L183-L196) and [queue](https://github.com/Azure/Azurite/blob/c9aa6fff44c6ec0e21f5f5aef8207790661719f3/src/queue/generated/utils/serializer.ts#L183-L196) serializers and presumably wants the same fix (a queue message XML body with multi-byte text looks like a candidate for the same corruption) — we only verified the table service.

As a workaround we currently patch the installed `serializer.js` in our CI test setup. Happy to open a PR with the fix.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with readRequestIntoText in src/table/generated/utils/serializer.ts and compare it with TableBatchUtils.StreamToString. Check the identical serializer helpers in src/blob/generated/utils/serializer.ts and src/queue/generated/utils/serializer.ts, then run the documented TCP-proxy repro. Done means split multi-byte UTF-8 values round-trip without U+FFFD corruption across the affected services.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
api, backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.