About convert MARCO dataset to Dureader style
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 306
- PR merge metrics
- No merged PRs in 30d
Description
When using the script marcov2_to_dureader.py to convert MARCOv2 to dureader, it failed because ValueError: Trailing data
The command: sh run_marco2dureader_preprocess.sh ../Marco/train_v2.1.json ../Marco/train_v2.1_dureaderformat.json
But it occurs an error -- ValueError: Trailing data. Details as follow:
Traceback (most recent call last):
File "marcov1_to_dureader.py", line 33, in
df = pd.read_json(sys.argv[1])
File "/home/user/anaconda3/lib/python3.6/site-packages/pandas/io/json/json.py", line 366, in read_json
return json_reader.read()
File "/home/user/anaconda3/lib/python3.6/site-packages/pandas/io/json/json.py", line 467, in read
obj = self._get_object_parser(self.data)
File "/home/user/anaconda3/lib/python3.6/site-packages/pandas/io/json/json.py", line 484, in _get_object_parser
obj = FrameParser(json, **kwargs).parse()
File "/home/user/anaconda3/lib/python3.6/site-packages/pandas/io/json/json.py", line 576, in parse
self._parse_no_numpy()
File "/home/user/anaconda3/lib/python3.6/site-packages/pandas/io/json/json.py", line 793, in _parse_no_numpy
loads(json, precise_float=self.precise_float), dtype=None)
ValueError: Trailing data
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with marcov2_to_dureader.py and run_marco2dureader_preprocess.sh using the command and MARCOv2 input shown in the report. Investigate why pandas.read_json raises ValueError: Trailing data, noting that the traceback names marcov1_to_dureader.py; done means the conversion completes and produces the requested DuReader-format output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100