load_stream loads the whole thing when used intuitively
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 597
- Forks
- 223
- Avg merge
- 11h 23m
- Merged PRs (30d)
- 3
Description
Surprisingly, load_stream doesn't handle streams but loads all docs in an IO object at once, and only then iterates on the result. This results in much latency as well as huge swaths of memory being taken hostage when used intuitively.
Maybe this could return an Enumerator (that could be to_a) or do a lazy map (that could be forced).
Example "naive" code:
# tested with ruby 2.2.3
require 'yaml'
def rss
`ps -p #{Process.pid} -o rss`.split("\n").last.to_i
end
unless File.exist? 'test.yml'
File.open('test.yml', 'wb') do |f|
1_000_000.times do
f.write "---\n"
f.write format("foo: '%016x'\n", Random.rand(2**64))
end
end
end
s = YAML.load_stream File.open('test.yml')
puts 'loaded'
s.each.with_index do |_, i|
puts i if i % 100_000 == 0
end
puts "rss: #{rss} kB"
While load_stream does support a block arg, doing the following is cumbersome and non-discoverable:
s = Enumerator.new do |y|
YAML.load_stream(File.open('test.yml')) { |doc| y << doc }
end
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review lib/psych.rb around the load_stream implementation and compare its block and non-block behavior using the large YAML example in the issue. Done means intuitive iteration begins processing documents without first loading the entire stream into memory, while the existing block usage remains usable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100