hydradatabase / hydradatabase/columnar
[Feature]: Please help support new feature to identify and fix the corrupted stripe in large hydra table
- Dominant language
- C
- Stars
- 3k
- Forks
- 106
- PR merge metrics
- No merged PRs in 30d
Description
### What's wrong?
Summary
Currently a decompression/read error on one corrupted stripe aborts reads, VACUUM and index builds for the whole columnar table (DecompressBuffer -> ereport(ERROR)). For very large tables this means a single corrupted stripe can take the whole table offline.
Expected / requested behavior
Please consider an opt-in mode that:
- detects a decompression/read failure for a stripe,
- records the stripe id as “corrupted” in metadata (visible via SQL),
- skips that stripe during reads and maintenance (VACUUM / index build) so the rest of the table continues to work,
- exposes a small admin API/view so operators can list/repair/clear corrupted stripes.
Why this helps
- Makes large tables more resilient: a single corrupted stripe won’t stop queries on the rest of the table, and operators get a clear list of affected stripes to repair.
Minimal implementation idea (optional)
- An opt-in GUC (e.g. columnar.ignore_corrupted_stripes = on) + a reader-side PG_TRY around decompression that logs the stripe and marks it in metadata, returning NULLs for that stripe’s rows.
Thanks
Thanks for the great work on this extension — happy to help sketch or prototype a minimal patch if useful.
Frank
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the reader path around DecompressBuffer -> ereport(ERROR), then follow how VACUUM and index builds process stripes. Done means an opt-in mode can record corrupted stripe IDs in SQL-visible metadata, skip those stripes during reads and maintenance, and provide admin operations to list, repair, and clear them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, postgresql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100