Dead fragments and version history accumulate with no way to reclaim them

During a crawl, every checkpoint flush is its own LanceDB commit, and update-style operations replace fragments while historical versions keep the old ones pinned. Nothing in the index lifecycle ever reclaims either. @davidh233's team hit the consequences while running scheduled offline indexing of large monorepos: a full build of their large repository ended at 1,342 fragments, and since a point lookup scans every fragment (#85), every later read pays for all of them. The dead versions also bloat the database on disk, which matters when the artifact is copied or archived.

#83 and #85 reduce how fast this accumulates. But any long-lived database still accumulates it, and the only remedy today is purge and a full re-crawl.

Investigation

Their team prototyped a cleanup command that compacts each table's fragments and then prunes all historical versions, run once after a successful crawl. Measured on two repositories:

Metric	Small repository	Large repository
Fragments	84 to 1	1,342 to 1
Database size	61.5 to 48.7MB	1.39 to 1.04GB
Same-commit re-crawl	70s to 14s	255s to 19s
Cleanup cost	under 1s	4s

The cleanup pays for itself on the first re-crawl after it, by a wide margin. Their version prunes with a zero retention window, dropping every version except the latest. For their pipeline that is correct: the database being cleaned is a freshly built artifact that nothing else has open, so no reader can be holding an older version.

Proposal

It seems like this can be solved with a single new command:

monodex gc            # compact fragments, prune old versions (keeps a retention window)
monodex gc --unsafe   # prune all history; requires that nothing else is using the database

1. Compact, then prune. For the chunks and label_metadata tables: merge small fragments, then drop historical versions and free the fragments they pinned. Same mechanism as the prototype.

2. The default keeps a retention window. Monodex readers are lock-free by design, so a search running concurrently with gc may be holding an older table version and needs to finish against it. That argues for a default that keeps recent history, which is the one behavioral difference from the prototype: a typical local database is live in a way a freshly built publishing artifact is not. This is not a novel design; it is what git gc does, where gc.pruneExpire defaults to two weeks for exactly the same reason. The window length is the part most worth arguing about. 24 hours is a starting guess, on the reasoning that searches are short-lived subprocesses so almost any window is generous, but if someone has a workload where a reader can hold a version open much longer, that changes the number.

3. --unsafe drops the window to zero. That is the prototype's behavior, and for an isolated database cleaned right before packaging it is the correct one. git gc --prune=now is the same flag under a different name, documented as safe only when nothing else is touching the repository. The precondition would be documented rather than enforced, because lock-free readers are unobservable and there is nothing the command could check. Rush uses --unsafe for a similar documented-precondition tradeoff in rush purge. Whether the flag should be named for the danger (--unsafe) or for the behavior (--prune-all, --full) is open.

4. Locking. Compaction and pruning rewrite the shared LanceDB dataset on behalf of every catalog, so gc is a database-spanning operation and takes the exclusive database lock, joining purge --all and init-db in the table in docs/design/concurrency.md. It would wait behind in-flight crawls with the standard progress messages rather than failing. Its LanceDB calls still acquire the commit mutex, since that rule holds without exception even for operations that could not contend.

5. Idempotent, with a report. A second run finds one fragment per table and no prunable history, and says so. Output reports fragments before and after, versions dropped, and bytes reclaimed.

Beyond --unsafe the proposal has no flags: no per-table selection, no compact-without-prune. That follows a general principle for Monodex, that a maintenance command people run on a schedule should be safe to run without arguments, and every option added is one more thing a reader of someone else's cron entry has to look up. It also assumes nobody actually wants those combinations, which is the kind of assumption worth contradicting if you have the case.

Why one verb rather than two

The obvious alternative is to call this optimize-db and leave a future cleanup command separate, on the grounds that they do different things. What makes one verb look better is where the command is going.

A database accumulates two kinds of leftovers: rows and folders Monodex itself no longer needs (chunks that lost their last label, FTS folders for labels that no longer exist, covered below), and the dead fragments and old versions measured above, which are files LanceDB leaves behind by design. Those are different mechanisms, but from the user's side they are the same complaint, that the database is bigger and slower than it should be, and the boundary between them is an implementation detail. They also have a natural order: deleting unneeded rows produces more dead data for the compaction pass to reclaim, so running only one cleanup finishes half the job. Two verbs make that ordering something users have to know. One verb keeps it internal. git gc made the same call, pruning unreachable objects and repacking under a single command.

The cost of one verb is that gc today would collect none of the Monodex-side garbage, so the name promises slightly more than it delivers until the sweeps below exist.

What gc could grow into

The Monodex-side half already has a backlog entry, BL52, which specifies three orphan kinds (chunk rows whose label membership is empty, vector payloads no retrieval selection references, and FTS folders for labels that no longer exist) and names monodex gc as the verb that sweeps them. That is where the name in this proposal comes from. Those sweeps would become additional phases running before the pass described here. Nothing here designs them; the point is that today's command should be shaped so they can land inside it later without a second verb.

Related

#83 and #85 reduce the rate at which fragments and versions accumulate; this issue reclaims what accumulates anyway. #84 would reduce version production further by batching per-file writes into per-batch writes.

BL52 is the Monodex-side counterpart described above, and the source of the gc name. BL104 reduces the version churn from label reassignment, which is another contributor to what this reclaims.

@davidh233, does a retention-window default create a problem for your pipeline, or is --unsafe enough for the packaging case?

During a crawl, every checkpoint flush is its own LanceDB commit, and update-style operations replace fragments while historical versions keep the old ones pinned. Nothing in the index lifecycle ever reclaims either. @davidh233's team hit the consequences while running scheduled offline indexing of large monorepos: a full build of their large repository ended at 1,342 fragments, and since a point lookup scans every fragment ([[#85](https://github.com/microsoft/monodex/issues/85)](https://github.com/microsoft/monodex/issues/85)), every later read pays for all of them. The dead versions also bloat the database on disk, which matters when the artifact is copied or archived.

[#83](https://github.com/microsoft/monodex/issues/83) and [#85](https://github.com/microsoft/monodex/issues/85) reduce how fast this accumulates. But any long-lived database still accumulates it, and the only remedy today is purge and a full re-crawl.

Investigation

Their team prototyped a cleanup command that compacts each table's fragments and then prunes all historical versions, run once after a successful crawl. Measured on two repositories:

Metric	Small repository	Large repository
Fragments	84 to 1	1,342 to 1
Database size	61.5 to 48.7MB	1.39 to 1.04GB
Same-commit re-crawl	70s to 14s	255s to 19s
Cleanup cost	under 1s	4s

The cleanup pays for itself on the first re-crawl after it, by a wide margin. Their version prunes with a zero retention window, dropping every version except the latest. For their pipeline that is correct: the database being cleaned is a freshly built artifact that nothing else has open, so no reader can be holding an older version.

Proposal

It seems like this can be solved with a single new command:

monodex gc            # compact fragments, prune old versions (keeps a retention window)
monodex gc --unsafe   # prune all history; requires that nothing else is using the database

1. Compact, then prune. For the chunks and label_metadata tables: merge small fragments, then drop historical versions and free the fragments they pinned. Same mechanism as the prototype.

2. The default keeps a retention window. Monodex readers are lock-free by design, so a search running concurrently with gc may be holding an older table version and needs to finish against it. That argues for a default that keeps recent history, which is the one behavioral difference from the prototype: a typical local database is live in a way a freshly built publishing artifact is not. This is not a novel design; it is what git gc does, where gc.pruneExpire defaults to two weeks for exactly the same reason. The window length is the part most worth arguing about. 24 hours is a starting guess, on the reasoning that searches are short-lived subprocesses so almost any window is generous, but if someone has a workload where a reader can hold a version open much longer, that changes the number.

3. --unsafe drops the window to zero. That is the prototype's behavior, and for an isolated database cleaned right before packaging it is the correct one. git gc --prune=now is the same flag under a different name, documented as safe only when nothing else is touching the repository. The precondition would be documented rather than enforced, because lock-free readers are unobservable and there is nothing the command could check. Rush uses --unsafe for a similar documented-precondition tradeoff in rush purge. Whether the flag should be named for the danger (--unsafe) or for the behavior (--prune-all, --full) is open.

4. Locking. Compaction and pruning rewrite the shared LanceDB dataset on behalf of every catalog, so gc is a database-spanning operation and takes the exclusive database lock, joining purge --all and init-db in the table in [docs/design/concurrency.md](https://github.com/microsoft/monodex/blob/main/docs/design/concurrency.md). It would wait behind in-flight crawls with the standard progress messages rather than failing. Its LanceDB calls still acquire the commit mutex, since that rule holds without exception even for operations that could not contend.

5. Idempotent, with a report. A second run finds one fragment per table and no prunable history, and says so. Output reports fragments before and after, versions dropped, and bytes reclaimed.

Beyond --unsafe the proposal has no flags: no per-table selection, no compact-without-prune. That follows a general principle for Monodex, that a maintenance command people run on a schedule should be safe to run without arguments, and every option added is one more thing a reader of someone else's cron entry has to look up. It also assumes nobody actually wants those combinations, which is the kind of assumption worth contradicting if you have the case.

Why one verb rather than two

The obvious alternative is to call this optimize-db and leave a future cleanup command separate, on the grounds that they do different things. What makes one verb look better is where the command is going.

A database accumulates two kinds of leftovers: rows and folders Monodex itself no longer needs (chunks that lost their last label, FTS folders for labels that no longer exist, covered below), and the dead fragments and old versions measured above, which are files LanceDB leaves behind by design. Those are different mechanisms, but from the user's side they are the same complaint, that the database is bigger and slower than it should be, and the boundary between them is an implementation detail. They also have a natural order: deleting unneeded rows produces more dead data for the compaction pass to reclaim, so running only one cleanup finishes half the job. Two verbs make that ordering something users have to know. One verb keeps it internal. git gc made the same call, pruning unreachable objects and repacking under a single command.

The cost of one verb is that gc today would collect none of the Monodex-side garbage, so the name promises slightly more than it delivers until the sweeps below exist.

What gc could grow into

The Monodex-side half already has a backlog entry, [BL52](https://github.com/microsoft/monodex/blob/main/docs/backlog.md#BL52), which specifies three orphan kinds (chunk rows whose label membership is empty, vector payloads no retrieval selection references, and FTS folders for labels that no longer exist) and names monodex gc as the verb that sweeps them. That is where the name in this proposal comes from. Those sweeps would become additional phases running before the pass described here. Nothing here designs them; the point is that today's command should be shaped so they can land inside it later without a second verb.

Related

[#83](https://github.com/microsoft/monodex/issues/83) and [#85](https://github.com/microsoft/monodex/issues/85) reduce the rate at which fragments and versions accumulate; this issue reclaims what accumulates anyway. [#84](https://github.com/microsoft/monodex/issues/84) would reduce version production further by batching per-file writes into per-batch writes.

[BL52](https://github.com/microsoft/monodex/blob/main/docs/backlog.md#BL52) is the Monodex-side counterpart described above, and the source of the gc name. [BL104](https://github.com/microsoft/monodex/blob/main/docs/backlog.md#BL104) reduces the version churn from label reassignment, which is another contributor to what this reclaims.

@davidh233, does a retention-window default create a problem for your pipeline, or is --unsafe enough for the packaging case?

Start with docs/design/concurrency.md and the existing purge --all and init-db command paths to understand database-wide locking and progress behavior. Then trace the chunks and label_metadata LanceDB operations and the search entry point. Done means a gc command compacts both tables, prunes history with a documented retention window or --unsafe zero window, behaves idempotently, and reports reclaimed work.