huggingface / huggingface/datatrove
Conflicting library requirements for datatrove==0.7.0
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
Using datatrove version 0.7.0 to run a job in Ray cluster, via RayPipelineExecutor.
Constructed a venv as per pyproject.toml, however facing a conflict between two libraries from the requirements.
Specifically, libraries multiprocess and numpy are conflicting.
The log of error is: AttributeError: module 'numpy' has no attribute 'AxisError'
Apparently, this is happening due to the fact that AxisError class has been moved in the numpy>=2.0.0 versions, whereas the latest multiprocess library works on the first major version of numpy, for example numpy==1.26.4
As the current resource removed multiprocess code from RayPipelineExecutor class to a simple Python for loop.
Wondering if this issue is happened only with me, and how properly resolve it?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by recreating the venv from pyproject.toml and running a RayPipelineExecutor job to confirm the AttributeError involving numpy.AxisError. Inspect the declared multiprocess and numpy versions and the RayPipelineExecutor usage; done means the dependency conflict has a verified resolution or a documented workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100