openstreetmap / openstreetmap/OSM-binary
Unable to distinguish valid stream end from broken stream
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 217
- Forks
- 79
- Avg merge
- 23h 31m
- Merged PRs (30d)
- 3
Description
Hi
I had recently run into an issue while importing large pbf file.
The problem was broken input stream and I ended up importing only half of the expected data that got unnoticed.
It's impossible to find out whether the import completed successfully and the EOFException is legit (basically that will be the case only while reading headersize in FileBlockHead.readHead) or whether the stream is broken in the middle of processing and it can not read the data to be expected down the line (any other place).
Either exception shall be thrown in such a case or another callback method in Sink introduced so Sink::complete() is called upon success and Sink:failure() upon error?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading FileBlockHead.readHead and the Sink interface or entry points involved in importing a large PBF stream. Determine how legitimate end-of-stream handling differs from failures during later processing, then define a completion or failure signal whose behavior can be tested for both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100