introspect/dump-cluster: parallelize per-node and per-database work
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 15
- Forks
- 2
- Avg merge
- 1h 1m
- Merged PRs (30d)
- 27
Description
Symptom
All introspection is single-threaded: dump-cluster iterates nodes sequentially with a fresh connection each (cmd/hclexp/hclexp.go:392-400), and introspectSchema loops databases serially (hclexp.go:271-280).
Impact
Dumping a large fleet is O(nodes x databases) wall-clock; a 30-node cluster dump takes 30x longer than it needs to.
Fix direction
Fan out per-node dumps (and optionally per-database introspection) with an errgroup and a bounded concurrency flag (e.g. -parallel 8). Output writing is already per-node files, so there is no shared-state hazard beyond logging.
Found in the 2026-07-02 deep-dive audit (docs/plans/2026-07-02-deep-dive-improvement-areas.md, B8).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read cmd/hclexp/hclexp.go:392-400 and introspectSchema around hclexp.go:271-280, then consult the audit note at docs/plans/2026-07-02-deep-dive-improvement-areas.md. Trace how dump-cluster creates per-node connections and files before selecting bounded errgroup concurrency. Done means node work is parallelized with a -parallel limit, per-node output remains separate, and optional per-database parallelism is handled without shared-state hazards.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, databases, tooling
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100