huggingface / huggingface/datatrove
task hang when extraction raises error
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
I implement another custom extractor and my job hangs when there is a bug during processing.
```bash
2025-02-10 06:51:50.966 | INFO | datatrove.pipeline.readers.base:read_files_shard:191 - Reading input file .json, 1/199
Process Process-1:
Traceback (most recent call last):
File "/opt/conda/envs/datatrove/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/envs/datatrove/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/output/datatrove/src/datatrove/pipeline/extractors/base.py", line 103, in _worker
extract_fn("") # "warmup"
File "/output/datatrove/src/datatrove/pipeline/extractors/magic_html.py", line 53, in extract
return self.magic_html_extractor_v1(
File "/output/datatrove/src/datatrove/pipeline/extractors/magic_html.py", line 37, in magic_html_extractor_v1
extracted_html = GeneralExtractor().extract(html=html, base_url=base_url).get("html", "")
File "/opt/conda/envs/datatrove/lib/python3.10/site-packages/magic_html/__init__.py", line 58, in extract
return ArticleExtractor().extract(html=html, **kwargs)
File "/opt/conda/envs/datatrove/lib/python3.10/site-packages/magic_html/extractors/article_extractor.py", line 16, in extract
raise ValueError
ValueError
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.