Azure / Azure/Azurite

[Blob] __blobstorage__ grows without bound: GC cannot reclaim partially-referenced extents

Open
#2,767 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
2.3k
Forks
393
Avg merge
1d 20h
Merged PRs (30d)
36

Description

### Which service(blob, file, queue, table) does this issue concern?

Blob (the queue service shares `FSExtentStore` and is affected in the same way).

### Which version of the Azurite was used?

3.37.0, and reproduced against `main`.

### Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

DockerHub in development; reproduced locally from a build of `main` (npm).

### What's the Node.js version?

v22.22.2 (Linux x64).

### What problem was encountered?

On a long-running Azurite container (~month, persistent volume, continuous
overwrite/delete churn), `__blobstorage__` grew to **~52 GB while the live blob
data was only ~500 MB**. The GC never reclaims it.

This is not GC lag and not a GC failure — the GC is behaving exactly as
designed. The space is structurally unreclaimable:

- `FSExtentStore.appendExtent()` packs many blobs into shared extent files of
up to `DEFAULT_MAX_EXTENT_SIZE` (64 MB, `src/common/utils/constants.ts`).
- `BlobGCManager.markSweep()` is whole-file mark-and-sweep. It builds a set of
all extent IDs, removes every ID that is still referenced, and deletes what
remains. There is no chunk-level accounting, so an extent is deleted only
when **every** reference into it is gone.
- `BlobReferredExtentsAsyncIterator` yields only `persistency.id`, discarding
`offset`/`count`, so per-extent live-byte usage is not computed anywhere.

Consequently **a single surviving 1 KB blob pins its entire 64 MB extent
forever**. Under sustained churn, long-lived blobs get scattered across many
extents that were mostly filled with data that has since been deleted, and disk
usage grows without bound. ~52 GB across ~830 extents at ~1% utilization each
is exactly what this predicts after a months.

The class comment on `BlobGCManager` already anticipates the missing piece:

> In the future, GC manager can also help merging small extent mapped files
> into one big file to improve the performance.

Measured on a build of `main` with the attached script — 10 rounds of "write
one 1 MB blob that is kept, then 56 MB of blobs written and deleted
immediately":

```
--- immediately after churn
extent files on disk : 9 (570.0 MB)
live referenced bytes : 10.0 MB
dead bytes pinned in referenced extents : 512.0 MB <-- never reclaimable
bytes in fully-unreferenced extents : 48.0 MB

--- after 22 minutes of GC running
extent files on disk : 9 (570.0 MB)
live referenced bytes : 10.0 MB
dead bytes pinned in referenced extents : 512.0 MB
bytes in fully-unreferenced extents : 48.0 MB

per-extent utilization:
28ff88b7 size=66.0 MB live=2.0 MB util=3.03%
545789d7 size=65.0 MB live=1.0 MB util=1.54%
247365e3 size=65.0 MB live=1.0 MB util=1.54%
86f669c4 size=65.0 MB live=1.0 MB util=1.54%
57dd5047 size=65.0 MB live=1.0 MB util=1.54%
839e2170 size=65.0 MB live=1.0 MB util=1.54%
39b06b69 size=65.0 MB live=1.0 MB util=1.54%
4c9020d5 size=66.0 MB live=2.0 MB util=3.03%
48932e93 size=48.0 MB live=0.0 MB util=0.00%
```

570 MB of disk holding 10 MB of live data. 22 minutes covers more than two full
mark-sweep cycles past the 10-minute `DEFAULT_EXTENT_GC_PROTECT_TIME_IN_MS`
window, and the GC deleted nothing — correctly, since the only fully
unreferenced extent (`48932e93`) is the active write extent that
`FSExtentStore.isActiveExtent()` deliberately skips.

The only workaround today is deleting `__blobstorage__`, which destroys the
live data along with the dead bytes.

### Steps to reproduce the issue?

Attached: `

[azurite-extent-fragmentation-repro.js](https://github.com/user-attachments/files/31737972/azurite-extent-fragmentation-repro.js)

`. It drives the churn through
the public blob API and then classifies every byte in the extent store as live,
dead-but-pinned, or reclaimable by reading Azurite's own metadata.

```
npm install @azure/storage-blob
npx azurite-blob --location ./azurite-data --silent &
node azurite-extent-fragmentation-repro.js ./azurite-data
```

`ROUNDS` and `WAIT_MINUTES` are configurable; the default 22-minute wait is
chosen to exceed the GC interval plus the extent protect time.

### Have you found a mitigation/solution?

Yes — extent compaction, built on top of the existing GC rather than replacing
it. I have a working prototype and would be glad to open a PR if the approach
looks acceptable.

The idea: identify extents whose live-byte ratio is below a threshold, copy
their live chunks out through the normal `appendExtent()` path, and rewrite the
blob metadata references to the new locations.

**The key design property is that compaction never deletes anything.** Once its
live chunks have been relocated, the emptied extent is simply unreferenced, and
the existing mark-and-sweep deletes it on a later pass — still behind the
protect window and the `isActiveExtent()` guard. That makes the concurrency
story tractable:

- in-flight reads against the old extent keep working, because the file is
still there;
- chunks shared between blobs (snapshots and `startCopyFromURL` both copy
`persistency` references) are safe — relocating only some references means
the extent just isn't reclaimed this round;
- if a blob is deleted or overwritten mid-relocation, the store re-reads the
document, sees the chunks no longer match, and skips the update; the
freshly written copy becomes ordinary garbage that the GC reclaims.

Additionally, each relocated chunk would be read back from its new location and
compared against the source before its metadata reference is updated, so a
faulty relocation costs disk space rather than corrupting a blob.

Prototype result on the data directory above — 570 MB to 58 MB, with all blobs
verified byte-identical after a restart (the residual 48 MB is the old active
write extent, which normal GC removes once it is no longer active):

```
before: 570.0 MB on disk, 9 extents, 10.0 MB live
selected 8 extents below 25% utilization
moved 10.0 MB of live data
after: 58.0 MB on disk, 2 extents
verified 10 blobs, 0 mismatches
```

Sketch of the change: a new `IBlobMetadataStore` method to relocate chunk
references (implemented for Loki and SQL — SQL only needs the `persistency` and
`committedBlocksInOrder` columns, since it does not persist page ranges), a new
`BlobExtentCompactor` under `src/blob/gc/`, a call site in `markSweep()`, and an
opt-in CLI flag so existing behaviour is unchanged by default. Extents
referenced by uncommitted blocks would be excluded from candidacy rather than
partially relocated.

Questions before I invest in a PR:

1. Is compaction in the GC the direction you would want, or would you prefer an
offline/maintenance-mode command that operates on a stopped data directory?
2. Should it be opt-in behind a flag, or on by default with a conservative
utilization threshold and a per-cycle byte budget?
3. Would you want the same treatment for the queue service in the same change,
or kept separate?

Two unrelated minor things noticed while investigating, happy to fold into a
separate PR:

- `LokiExtentMetadataStore.updateExtent()` writes `lastModifiedInMS` on insert
but `LastModifyInMS` on update, so a freshly inserted extent does not match
the protect-time predicate in `listExtents()` in the intended way.
- `BlobGCManager.markSweep()` logs `allExtents.entries` (a function reference)
where the count was intended.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/common/utils/constants.ts, BlobGCManager.markSweep(), and BlobReferredExtentsAsyncIterator, then run the attached azurite-extent-fragmentation-repro.js against a persistent data directory. Review the proposed IBlobMetadataStore change and the new src/blob/gc/BlobExtentCompactor entry point; done means compacted live chunks remain byte-identical, stale extents become reclaimable, and existing GC behavior remains safe.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, sql, typescript
Domain
backend, databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.