huggingface / huggingface/datatrove

JsonlWriter failed when trying to write Timestamp object

Open
#325 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

Hi,
Thanks for this great library

I just came across a little bug.

When working on the [bigcode/the-stack-v2-train-full-ids](https://huggingface.co/datasets/bigcode/the-stack-v2-train-full-ids) there are some columns with timestamp that are automatically map when read with the `ParquetReader` but can't be write directly by the `JsonlWriter`

Example :
```
'metadata': {
'repo_name': 'bryandito/kitchen',
'repo_url': 'https://github.com/bryandito/kitchen',
'snapshot_id': 'b5ac726477b5712a8f32c83ed27ea72f797784f2',
'revision_id': '47d42b540bd42a3b7fcd9bf77c310a3f7d8a2eb1',
'directory_id': '4b35f1fab0cce7d6df02c42c7e00232e73b8f7ff',
'branch_name': 'refs/heads/master',
'visit_date': Timestamp('2021-05-04 22:44:02.776767'),
'revision_date': Timestamp('2018-02-06 23:00:56'),
'committer_date': Timestamp('2018-02-06 23:00:56')
}
```
Fails with :
```
TypeError: Type is not JSON serializable: Timestamp
```
with pipeline :
```
pipeline=[
ParquetReader(
f"hf://datasets/bigcode/the-stack-v2-train-full-ids/data/",
glob_pattern="*.parquet",
text_key="files"
),
JsonlWriter(output_path, compression=None)
],
```

A small fix to do,
Best regards and thanks again for your commitment to open-source

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the JsonlWriter used in the ParquetReader-to-JsonlWriter pipeline and reproduce the failure using the Timestamp values shown in the issue. Done means the pipeline writes records containing Timestamp objects without the reported JSON serialization TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.