dmlc / dmlc/xgboost

Support for Spark version >= 3.1 and AssertionError query

Open
#7,310 13 comments 0 reactions 0 assignees View on GitHub
status: need update
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

Hi!
I wanted to confirm if XGBoost supports Spark version 3.1.2. I have been trying to run XGBoost on the latest version of Apache Spark on a dataset > 3TB on a 28 node cluster.

Also, I have been getting the following error and haven't been able to figure out what might be causing this.
```
21/10/10 10:50:17 INFO XGBoostSpark: starting training with timeout set as 1800000 ms for waiting for resources
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: Exception in thread Thread-1:
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: Traceback (most recent call last):
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: self.run()
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: File "/usr/lib/python2.7/threading.py", line 754, in run
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: self.__target(*self.__args, **self.__kwargs)
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: File "/tmp/tracker2488390078882395804.py", line 324, in run
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: self.accept_slaves(nslave)
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: File "/tmp/tracker2488390078882395804.py", line 268, in accept_slaves
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: s = SlaveEntry(fd, s_addr)
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: File "/tmp/tracker2488390078882395804.py", line 64, in __init__
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: assert magic == kMagic, 'invalid magic number=%d from %s' % (magic, self.host)
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: AssertionError: invalid magic number=542393671 from 143.198.38.116
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger:
21/10/10 13:44:27 INFO RabitTracker$TrackerProcessLogger: Tracker Process ends with exit code 0
```

I look forward to hearing back from you. Thank you in advance for your help and time.

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file, test, or entry point is identified. Start by locating the Spark integration and RabitTracker implementation, then reproduce the reported run with Spark 3.1.2 and the supplied AssertionError details. Done means establishing whether that Spark version is supported and documenting or fixing the cause of the invalid magic number.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.