huggingface / huggingface/datatrove

Conflicting library requirements for datatrove==0.7.0

Open
#443 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

Using datatrove version 0.7.0 to run a job in Ray cluster, via RayPipelineExecutor.
Constructed a venv as per pyproject.toml, however facing a conflict between two libraries from the requirements.
Specifically, libraries multiprocess and numpy are conflicting.
The log of error is: AttributeError: module 'numpy' has no attribute 'AxisError'

Apparently, this is happening due to the fact that AxisError class has been moved in the numpy>=2.0.0 versions, whereas the latest multiprocess library works on the first major version of numpy, for example numpy==1.26.4

As the current resource removed multiprocess code from RayPipelineExecutor class to a simple Python for loop.
Wondering if this issue is happened only with me, and how properly resolve it?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by recreating the venv from pyproject.toml and running a RayPipelineExecutor job to confirm the AttributeError involving numpy.AxisError. Inspect the declared multiprocess and numpy versions and the RayPipelineExecutor usage; done means the dependency conflict has a verified resolution or a documented workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.