huggingface / huggingface/faceberg
Bug: `write_manifest` reads `to_manifest_file()` before the closure of the context manager, leading to incorrect writing of metadata
- Dominant language
- Python
- Stars
- 31
- Forks
- 2
- Avg merge
- 23m
- Merged PRs (30d)
- 1
Description
In the iceberg module, the [write_manifest](https://github.com/huggingface/faceberg/blob/2346ee32ba68a7f906e1244d4b9056c404f35722/faceberg/iceberg.py#L366) is currently called inside the with `_write_manifest(...)` block. At that point, the manifest writer is still open and may not have finished flushing the output file. This leads to malformed or incorrect metadata written. Move [manifest = writer.to_manifest_file()](https://github.com/huggingface/faceberg/blob/2346ee32ba68a7f906e1244d4b9056c404f35722/faceberg/iceberg.py#L459) outside the with block. This ensures the writer has finished writing and closing the file before its metadata is read. For reproduction of the error, please check the [re-implemented version](https://github.com/ParagEkbote/hf-genomic-dataset-enrichment/blob/fce3d149c16d6613d061b4af250e0ef6b2841483/carbon-enrichment/carbon_enrichment/resources/faceberg.py#L128) of this method which fixes the issue for a data enrichment use-case. WDYT?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in faceberg/iceberg.py at write_manifest and inspect the _write_manifest context manager, especially the manifest = writer.to_manifest_file() call. Compare the method with the linked re-implemented version to understand the reported ordering. Done means metadata is read only after the writer context has closed, with the malformed-writing reproduction no longer occurring.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100