[Request] can dwalk "stream" text output as it reads mfu file (to avoid high RAM usge)?
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 200
- Forks
- 85
- Avg merge
- 3d 21h
- Merged PRs (30d)
- 2
Description
I have a large 1.4TB .mfu file generated by dwalk for 502M items.
I want to generate an unsorted text output file from this mfu file.
Does dwalk read the entire mfu file into RAM before outputting the text file?
For sorted output, I could see reading into all into RAM. But for unsorted output, could dwalk “stream” the output as it reads the mfu input and thereby not use much ram?
I’m asking because I have a service node that can generate the generate the mfu file but doesn’t have enough ram to generate (unsorted) text output from that same mfu file.
Thanks!
- Mark
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing dwalk's conversion from an .mfu input to unsorted text output and determine whether records are buffered in memory. Done means the unsorted output is produced incrementally without requiring the full 1.4 TB file in RAM, with the behavior validated on a large input.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- cli, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100