baidu / baidu/DuReader

About convert MARCO dataset to Dureader style

Open
#70 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
306
PR merge metrics
No merged PRs in 30d

Description

When using the script marcov2_to_dureader.py to convert MARCOv2 to dureader, it failed because ValueError: Trailing data

The command: sh run_marco2dureader_preprocess.sh ../Marco/train_v2.1.json ../Marco/train_v2.1_dureaderformat.json

But it occurs an error -- ValueError: Trailing data. Details as follow:
Traceback (most recent call last):
File "marcov1_to_dureader.py", line 33, in
df = pd.read_json(sys.argv[1])
File "/home/user/anaconda3/lib/python3.6/site-packages/pandas/io/json/json.py", line 366, in read_json
return json_reader.read()
File "/home/user/anaconda3/lib/python3.6/site-packages/pandas/io/json/json.py", line 467, in read
obj = self._get_object_parser(self.data)
File "/home/user/anaconda3/lib/python3.6/site-packages/pandas/io/json/json.py", line 484, in _get_object_parser
obj = FrameParser(json, **kwargs).parse()
File "/home/user/anaconda3/lib/python3.6/site-packages/pandas/io/json/json.py", line 576, in parse
self._parse_no_numpy()
File "/home/user/anaconda3/lib/python3.6/site-packages/pandas/io/json/json.py", line 793, in _parse_no_numpy
loads(json, precise_float=self.precise_float), dtype=None)
ValueError: Trailing data

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with marcov2_to_dureader.py and run_marco2dureader_preprocess.sh using the command and MARCOv2 input shown in the report. Investigate why pandas.read_json raises ValueError: Trailing data, noting that the traceback names marcov1_to_dureader.py; done means the conversion completes and produces the requested DuReader-format output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.