ruby / ruby/psych

load_stream loads the whole thing when used intuitively

Open
#259 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
597
Forks
223
Avg merge
11h 23m
Merged PRs (30d)
3

Description

Surprisingly, load_stream doesn't handle streams but loads all docs in an IO object at once, and only then iterates on the result. This results in much latency as well as huge swaths of memory being taken hostage when used intuitively.

Maybe this could return an Enumerator (that could be to_a) or do a lazy map (that could be forced).

Example "naive" code:

# tested with ruby 2.2.3
require 'yaml'

def rss
  `ps -p #{Process.pid} -o rss`.split("\n").last.to_i
end

unless File.exist? 'test.yml'
  File.open('test.yml', 'wb') do |f|
    1_000_000.times do
      f.write "---\n"
      f.write format("foo: '%016x'\n", Random.rand(2**64))
    end
  end
end

s = YAML.load_stream File.open('test.yml')
puts 'loaded'

s.each.with_index do |_, i|
  puts i if i % 100_000 == 0
end

puts "rss: #{rss} kB"

While load_stream does support a block arg, doing the following is cumbersome and non-discoverable:

s = Enumerator.new do |y|
  YAML.load_stream(File.open('test.yml')) { |doc| y << doc }
end

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review lib/psych.rb around the load_stream implementation and compare its block and non-block behavior using the large YAML example in the issue. Done means intuitive iteration begins processing documents without first loading the entire stream into memory, while the existing block usage remains usable.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.