infra/helper: fix false success and partial corpus updates when public corpus extraction fails
- Dominant language
- Shell
- Stars
- 12.6k
- Forks
- 2.9k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 62
Description
### Summary
The public-corpus download workflow in `infra/helper.py` has two failure-handling problems:
1. If `unzip` cannot be executed, the helper can report success without extracting any inputs.
2. If extraction fails partway through an archive, existing local corpus files may already have been overwritten.
These behaviors were checked against commit [`[1d2405c](https://github.com/google/oss-fuzz/commit/1d2405cfbf4e7819cb5d5a73aa8dfc5b55894e35)`](https://github.com/google/oss-fuzz/commit/1d2405cfbf4e7819cb5d5a73aa8dfc5b55894e35).
### Current behavior
The relevant functions are [`_get_latest_public_corpus()` and `download_corpora()`](https://github.com/google/oss-fuzz/blob/1d2405cfbf4e7819cb5d5a73aa8dfc5b55894e35/infra/helper.py#L1051-L1144).
**Missing extraction executable**
`_get_latest_public_corpus()` catches `OSError` from the extraction subprocess and logs an error, but then removes the downloaded archive and returns `True`.
The per-target wrapper in `download_corpora()` also returns `True` whenever no exception escapes; it does not check the inner function’s return value.
**Partially failed extraction**
The helper runs `unzip -q -o` directly against the existing corpus directory. If a later archive entry fails its checksum, earlier entries may already have overwritten local files. The command reports failure, but the destination has been modified.
### Reproduction
These cases can be reproduced with isolated tests, without accessing cloud storage:
1. Substitute a valid local ZIP for the download and make the extraction subprocess raise `FileNotFoundError`, simulating missing `unzip`.
* Observed: `download_corpora()` returns `True`, mapping to exit code `0`, despite no extraction occurring.
2. Create an existing local seed and a ZIP containing a replacement for it, followed by an entry with an invalid CRC.
* Run extraction using the real `unzip` executable.
* Observed: `download_corpora()` returns `False`, but the existing seed has already been overwritten.
These are isolated helper-level checks, not a full OSS-Fuzz integration run.
### Expected behavior
* Download or extraction failures propagate to the command’s failure status.
* Archive download/extraction failures leave existing corpus inputs unchanged.
* Temporary artifacts are cleaned up consistently.
* Successful imports preserve unrelated local corpus inputs.
### Proposed approach
I would like to address this incrementally:
1. **Failure propagation:** fix extraction-error handling and add regression tests for success, missing executables, and subprocess failures.
2. **Staged extraction:** download and extract into a temporary workspace, merging into the destination only after successful extraction. Add tests for invalid archives, checksum failures, cleanup, and preservation of existing inputs.
3. **Workflow coverage and documentation:** test failure propagation into local coverage and Introspector workflows and document the behavior.
Staged extraction would protect against download and archive-extraction failures; it would not, by itself, guarantee an atomic final merge against disk failures or concurrent writers.
### Scope
This proposal is limited to the local public-corpus workflow. It does not change cloud bucket permissions, private-corpus download semantics, or CIFuzz’s optional-corpus behavior.
Related: #14334, which requests stronger regression coverage for helper CLI interactions.
Contributor guide
Research direction
Start in infra/helper.py with _get_latest_public_corpus() and download_corpora(), then reproduce the missing-unzip and invalid-CRC cases using isolated local ZIP files. Add regression coverage for failure propagation, staged extraction, cleanup, and preservation of existing corpus files; done means failures return a failure status without changing existing inputs, while successful imports retain unrelated inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100