ruby / ruby/json

Can `ResumableParser` return whether the last `partial_value` container is completed or not?

Open
#1,041 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
782
Forks
383
Avg merge
1d 18h
Merged PRs (30d)
10

Description

This is not a critical feature request.

I'm trying to migrate to json gem from json-stream gem in https://github.com/groonga/groonga-command-parser/ by JSON::ResumableParser.

It parses records to be loaded to Groonga (a full text search engine with column store). The records use the following format:

[
{"column_name1": "record1's column_value1", "column_name2": 2, ...},
{"column_name1": "record2's column_value1", "column_name2": 2, ...},
...
]

groonga-command-parser wants to process a record as soon as it's completed. For example, it wants to process the first record when the following chunk is processed:

[
{"column_name1": "record1's column_value1", "column_name2": 2, ...},

We can do it by the following script:

require "json"

parser = JSON::ResumableParser.new
parser << <<-CHUNK
[
{"column1": "value1"},
CHUNK
parser.parse
p parser.partial_value[0] # => {"column1" => "value1"}

But it doesn't work with the following chunk:

[
{"column_name1": "record1's column_value1",

We can't know whether the record is completed or not:

require "json"

parser = JSON::ResumableParser.new
parser << <<-CHUNK
[
{"column1": "value1",
CHUNK
parser.parse
p parser.partial_value[0] # => {"column1" => "value1"} # This is not completed yet

groonga-command-parser has a workaround for it. It ignores the last record:

require "json"

parser = JSON::ResumableParser.new
parser << <<-CHUNK
[
{"column1": "value1"},
CHUNK
parser.parse
p parser.partial_value[0..-2] # => []

parser << <<-CHUNK
{"column1": "value2"},
CHUNK
parser.parse
p parser.partial_value[0..-2] # => [{"column1" => "value1"}] # 1st record

If we can detect whether a record in this example is completed or not, it's useful. But this is not a critical feature request because there is a workaround for this case.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the two JSON::ResumableParser chunk examples in the issue and inspect how partial_value is populated after parse. Trace the parser state for a completed versus incomplete last record; done means callers can reliably distinguish those cases, including the workaround scenarios described.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.