huggingface / huggingface/datatrove

Bug: Default Adapter assumes type of metadata column in source data

Open
#328 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

In the last line below, `data.pop("metadata")` could be of type other than `dict`, and will fail then.

File: `src/datatrove/pipeline/readers/base.py`

```python

def _default_adapter(self, data: dict, path: str, id_in_file: int | str):
"""
The default data adapter to adapt input data into the datatrove Document format

Args:
data: a dictionary with the "raw" representation of the data
path: file path or source for this sample
id_in_file: its id in this particular file or source

Returns: a dictionary with text, id, media and metadata fields

"""
return {
"text": data.pop(self.text_key, ""),
"id": data.pop(self.id_key, f"{path}/{id_in_file}"),
"media": data.pop("media", []),
"metadata": data.pop("metadata", {}) | data, # remaining data goes into metadata
}
```

It happened when I tried to tokenize [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath), which has a `metadata` column with a a default string type.

```
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/multiprocess/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/local/lib/python3.10/dist-packages/datatrove/executor/local.py", line 76, in _launch_run_for_rank
return self._run_for_rank(rank, local_rank)
File "/usr/local/lib/python3.10/dist-packages/datatrove/executor/base.py", line 109, in _run_for_rank
raise e
File "/usr/local/lib/python3.10/dist-packages/datatrove/executor/base.py", line 90, in _run_for_rank
pipelined_data = pipeline_step(pipelined_data, rank, self.world_size)
File "/usr/local/lib/python3.10/dist-packages/datatrove/pipeline/base.py", line 119, in __call__
return self.run(data, rank, world_size)
File "/usr/local/lib/python3.10/dist-packages/datatrove/pipeline/tokens/tokenizer.py", line 390, in run
outputfile: TokenizedFile = self.write_unshuffled(data, unshuf_filename)
File "/usr/local/lib/python3.10/dist-packages/datatrove/pipeline/tokens/tokenizer.py", line 359, in write_unshuffled
for batch in batched(data, self.batch_size):
File "/usr/local/lib/python3.10/dist-packages/datatrove/utils/batching.py", line 20, in batched
while batch := list(itertools.islice(it, n)):
File "/usr/local/lib/python3.10/dist-packages/datatrove/pipeline/readers/huggingface.py", line 125, in run
document = self.get_document_from_dict(line, self.dataset, f"{rank:05d}/{li}")
File "/usr/local/lib/python3.10/dist-packages/datatrove/pipeline/readers/huggingface.py", line 60, in get_document_from_dict
document = super().get_document_from_dict(data, source_file, id_in_file)
File "/usr/local/lib/python3.10/dist-packages/datatrove/pipeline/readers/base.py", line 79, in get_document_from_dict
parsed_data = self.adapter(data, source_file, id_in_file)
File "/usr/local/lib/python3.10/dist-packages/datatrove/pipeline/readers/base.py", line 65, in _default_adapter
"metadata": data.pop("metadata", {}) | data, # remaining data goes into metadata
TypeError: unsupported operand type(s) for |: 'str' and 'dict'
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/datatrove/pipeline/readers/base.py at _default_adapter and trace how get_document_from_dict passes source records into it. Reproduce the failure with a record whose metadata value is a string, such as the FineMath data described in the issue. Done means the default adapter no longer raises this TypeError for non-dict metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.