huggingface / huggingface/faceberg

Bug: `write_manifest` reads `to_manifest_file()` before the closure of the context manager, leading to incorrect writing of metadata

Open Beginner friendly
#10 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
31
Forks
2
Avg merge
23m
Merged PRs (30d)
1

Description

In the iceberg module, the [write_manifest](https://github.com/huggingface/faceberg/blob/2346ee32ba68a7f906e1244d4b9056c404f35722/faceberg/iceberg.py#L366) is currently called inside the with `_write_manifest(...)` block. At that point, the manifest writer is still open and may not have finished flushing the output file. This leads to malformed or incorrect metadata written. Move [manifest = writer.to_manifest_file()](https://github.com/huggingface/faceberg/blob/2346ee32ba68a7f906e1244d4b9056c404f35722/faceberg/iceberg.py#L459) outside the with block. This ensures the writer has finished writing and closing the file before its metadata is read. For reproduction of the error, please check the [re-implemented version](https://github.com/ParagEkbote/hf-genomic-dataset-enrichment/blob/fce3d149c16d6613d061b4af250e0ef6b2841483/carbon-enrichment/carbon_enrichment/resources/faceberg.py#L128) of this method which fixes the issue for a data enrichment use-case. WDYT?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in faceberg/iceberg.py at write_manifest and inspect the _write_manifest context manager, especially the manifest = writer.to_manifest_file() call. Compare the method with the linked re-implemented version to understand the reported ordering. Done means metadata is read only after the writer context has closed, with the malformed-writing reproduction no longer occurring.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.