hyperdb-bootstrap never prunes superseded hyperd pins (793 MB for one active version)

Open
#299 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
powershell, rust

Research direction

Start at hyperdb-bootstrap, especially download_and_extract and refresh_current, then inspect the make clean target and build.ps1 equivalent. Clarify whether cleanup should retain one previous pin or only the active pin, and verify that installation and targeted cleanup remove stale versioned directories while preserving current/ and the active release.

Written by the indexing model from the issue text.

Description

Summary

hyperdb-bootstrap never removes a superseded hyperd release, so every pin bump leaves the previous one on disk permanently. On this workstation that is already 793 MB for a single pinned version:

263M	.hyperd/0.0.26359.r07abb490    <- superseded pin, nothing references it
265M	.hyperd/0.0.26479.r96880f6a    <- the current pin
265M	.hyperd/current                 <- a full copy of the current pin
793M	.hyperd

Two things compound:

  1. No pruning on bump. download_and_extract clears only the directory for the release it is installing, and refresh_current clears only current/. Neither considers older versioned directories, so each bump adds ~263 MB that is never reclaimed.
  2. current/ is a full file copy rather than a symlink. That is deliberate — it avoids requiring symlink privileges on Windows — but it means the active pin is always stored twice.

Nothing in the repository references the superseded directory by path; the only 26359 matches are version strings in changelogs, benchmark history, and a --version example.

Why the existing cleanup does not help

make clean does rm -rf .hyperd, which removes current/ along with everything else and forces a full re-download of the pinned release. So the only available cleanup is all-or-nothing: either carry the superseded copies indefinitely, or discard the working engine too and re-fetch ~265 MB.

There is no targeted way to say "keep the current pin, drop the rest."

Fix direction

Either would resolve it, and they compose:

  1. Prune on install. After refresh_current succeeds, remove versioned directories under .hyperd/ other than the one just installed. Doing it after the new pin is verified and current/ is refreshed keeps the operation safe — a failed download must not leave the tree with no usable engine.
  2. Add a targeted cleanup target. A make clean-hyperd-stale (plus the build.ps1 equivalent) that removes superseded versioned directories while leaving current/ and the active pin intact. Useful independently of (1), since it also cleans up trees that predate the change.

Worth considering whether to keep exactly one previous pin, which would make an A/B benchmark against the prior release — the procedure the update-hyperd-release skill documents — possible without re-downloading. That argues for pruning to N most recent rather than to one.

Provenance

Noticed while correcting the HYPERD_PATH guidance (#298). Verified against main at 05e993f.

Dominant language
Rust
Stars
2
Forks
2
Avg merge
12h 2m
Merged PRs (30d)
60

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from tableau/hyper-api-rust

All issues in tableau/hyper-api-rust

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.