Deadlock seen in MiGzOutputStream#close:329 version 1.0.0
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 84
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
Hi guys, first of all major thanks for open sourcing this library! I've been able to significantly improve the I/O performance of my application with this.
I just came across an instance where calling `MiGzOutputStream#close` hangs indefinitely. A heap dump reveals the thread in question is waiting for the thread pool to shut down.
https://github.com/linkedin/migz/blob/30730b6a85ce5678ed5e961f76bbea43da81bbbf/migz/src/main/java/com/linkedin/migz/MiGzOutputStream.java#L329
Is this a known issue? Is there any reason the ForkJoinPool introduced in 2.0-beta1 prevents this behavior?
Inputs:
- numThreads => 96 (48 core machine)
- outputStream => `FileOutputStream("/dev/null")`
- bytesIn => ~12GiB
- bytesOut => ~3GiB
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with migz/src/main/java/com/linkedin/migz/MiGzOutputStream.java around line 329 and inspect the close path and its thread-pool shutdown behavior. Reproduce with 96 threads, a FileOutputStream targeting /dev/null, and the reported input sizes, then compare the behavior involving the ForkJoinPool introduced in 2.0-beta1. Done means the reported close hang is explained and no longer occurs under those inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100