huggingface / huggingface/datatrove

task hang when extraction raises error

Open
#339 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

I implement another custom extractor and my job hangs when there is a bug during processing.
```bash
2025-02-10 06:51:50.966 | INFO | datatrove.pipeline.readers.base:read_files_shard:191 - Reading input file .json, 1/199
Process Process-1:
Traceback (most recent call last):
File "/opt/conda/envs/datatrove/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/envs/datatrove/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/output/datatrove/src/datatrove/pipeline/extractors/base.py", line 103, in _worker
extract_fn("") # "warmup"
File "/output/datatrove/src/datatrove/pipeline/extractors/magic_html.py", line 53, in extract
return self.magic_html_extractor_v1(
File "/output/datatrove/src/datatrove/pipeline/extractors/magic_html.py", line 37, in magic_html_extractor_v1
extracted_html = GeneralExtractor().extract(html=html, base_url=base_url).get("html", "")
File "/opt/conda/envs/datatrove/lib/python3.10/site-packages/magic_html/__init__.py", line 58, in extract
return ArticleExtractor().extract(html=html, **kwargs)
File "/opt/conda/envs/datatrove/lib/python3.10/site-packages/magic_html/extractors/article_extractor.py", line 16, in extract
raise ValueError
ValueError
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.