Azure / Azure/Azurite

Docker container terminates due to 'Invalid string length' error

Open
#2,539 6 comments 0 reactions 1 assignee Claimed by @blueww View on GitHub
table-storage
Dominant language
TypeScript
Stars
2.3k
Forks
393
Avg merge
1d 20h
Merged PRs (30d)
36

Description

Hi, I'm trying to insert around 7 million rows of fake data inside of a table storage resource. The rows consists only of several columns and they don't include any complex structures. I'm inserting the data through transactions, each containing 100 `Add` actions. The data is generated through `Bogus` & I've set sensible values for the `PartitionKey` & `RowKey` columns. `Azurite` is spun up as a `Docker` container w/ persistent storage.

Just below the 1 million row count the container terminates due to a runtime error from `/opt/azurite/node_modules/lokijs/src/lokijs.js:1331`:

```powershell
/opt/azurite/node_modules/lokijs/src/lokijs.js:1331
case "normal": return JSON.stringify(this, this.serializeReplacer);
^

RangeError: Invalid string length
at JSON.stringify ()
at Loki.serialize (/opt/azurite/node_modules/lokijs/src/lokijs.js:1331:36)
at Loki.saveDatabaseInternal (/opt/azurite/node_modules/lokijs/src/lokijs.js:2780:66)
at Loki.saveDatabase (/opt/azurite/node_modules/lokijs/src/lokijs.js:2822:12)
at Timeout.autosaveHandleInterval [as _onTimeout] (/opt/azurite/node_modules/lokijs/src/lokijs.js:2923:16)
at listOnTimeout (node:internal/timers:594:17)
at process.processTimers (node:internal/timers:529:7)
```

Going through the container logs I'm not seeing any other errors or warnings, all previous batches resolve with `HTTP 202` status codes.

It doesn't seem to happen at random, it's always around the 930k record mark, up until that it's reasonably fast. I've also looked at the recent issues in https://github.com/techfort/LokiJS, but I've yet to find any similar experiences.

I've also verified that I'm not using any in-memory storage, so I'm not sure what causes this dependency to error out. Also, I've reviewed the data in the transaction batch which causes this, there's nothing out of the ordinary. Artificial delays also don't resolve this issue.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.