huggingface / huggingface/datatrove
JsonlWriter failed when trying to write Timestamp object
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
Hi,
Thanks for this great library
I just came across a little bug.
When working on the [bigcode/the-stack-v2-train-full-ids](https://huggingface.co/datasets/bigcode/the-stack-v2-train-full-ids) there are some columns with timestamp that are automatically map when read with the `ParquetReader` but can't be write directly by the `JsonlWriter`
Example :
```
'metadata': {
'repo_name': 'bryandito/kitchen',
'repo_url': 'https://github.com/bryandito/kitchen',
'snapshot_id': 'b5ac726477b5712a8f32c83ed27ea72f797784f2',
'revision_id': '47d42b540bd42a3b7fcd9bf77c310a3f7d8a2eb1',
'directory_id': '4b35f1fab0cce7d6df02c42c7e00232e73b8f7ff',
'branch_name': 'refs/heads/master',
'visit_date': Timestamp('2021-05-04 22:44:02.776767'),
'revision_date': Timestamp('2018-02-06 23:00:56'),
'committer_date': Timestamp('2018-02-06 23:00:56')
}
```
Fails with :
```
TypeError: Type is not JSON serializable: Timestamp
```
with pipeline :
```
pipeline=[
ParquetReader(
f"hf://datasets/bigcode/the-stack-v2-train-full-ids/data/",
glob_pattern="*.parquet",
text_key="files"
),
JsonlWriter(output_path, compression=None)
],
```
A small fix to do,
Best regards and thanks again for your commitment to open-source
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the JsonlWriter used in the ParquetReader-to-JsonlWriter pipeline and reproduce the failure using the Timestamp values shown in the issue. Done means the pipeline writes records containing Timestamp objects without the reported JSON serialization TypeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100