Access violation in tree::ColMaker::Builder::EnumerateSplit() on Windows
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
I am getting the same error in XGBoost: tried versions 1.0 and 1.2. I never got such an error with a less parallelized processor. The processor on which I started to get the error is Ryzen Threadripper 3990X (64 physical, 128 logical cores). The call stack is:
```
> xgboost.dll!xgboost::tree::ColMaker::Builder::EnumerateSplit(const xgboost::Entry * begin, const xgboost::Entry * end, int d_step, unsigned int fid, const std::vector,std::allocator>> & gpair, std::vector> & temp) Line 395 C++
xgboost.dll!xgboost::tree::ColMaker::Builder::UpdateSolution$omp$1() Line 456 C++
vcomp140.dll!__vcomp_fork_helper() Line 91 Unknown
vcomp140.dll!_vcomp::fork_helper_wrapper(void(*)() funclet, int arg_count, char * argptr) Line 348 C++
vcomp140.dll!_vcomp::ParallelRegion::HandlerThreadFunc(void * context, unsigned long index) Line 329 C++
vcomp140.dll!_vcomp::PersistentThreadFunc(void * pvContext) Line 240 C++
kernel32.dll!BaseThreadInitThunk() Unknown
ntdll.dll!RtlUserThreadStart() Unknown
```
The lines that throw are:
```
for (i = 0, p = it; i < kBuffer; ++i, p += d_step) {
buf_position[i] = position_[p->index];
buf_gpair[i] = gpair[p->index]; // The debugger arraw is pointing here
}
```
The error is:
> Exception thrown at 0x00007FFD6E459706 (xgboost.dll) in BatchXgbTrainClose.exe: 0xC0000005: Access violation reading location 0x0000016652B2D930.
If the debugger shows it to me correctly, the value of `p->index` is 1074375450 .
The full call stack of the main (i.e. not an OMP) thread that led here is:
```
> xgboost.dll!xgboost::tree::ColMaker::Builder::EnumerateSplit(const xgboost::Entry * begin, const xgboost::Entry * end, int d_step, unsigned int fid, const std::vector,std::allocator>> & gpair, std::vector> & temp) Line 399 C++
xgboost.dll!xgboost::tree::ColMaker::Builder::UpdateSolution$omp$1() Line 456 C++
vcomp140.dll!__vcomp_fork_helper() Line 91 Unknown
vcomp140.dll!_vcomp::fork_helper_wrapper(void(*)() funclet, int arg_count, char * argptr) Line 348 C++
vcomp140.dll!_vcomp::ParallelRegion::HandlerThreadFunc(void * context, unsigned long index) Line 329 C++
vcomp140.dll!InvokeThreadTeam(_THREAD_TEAM * ptm, void(*)(void *, unsigned long) pvContext, void *) Line 842 C++
vcomp140.dll!_vcomp_fork(int if_test, int arg_count, void(*)() funclet, ...) Line 230 C++
xgboost.dll!xgboost::tree::ColMaker::Builder::UpdateSolution(const xgboost::SparsePage & batch, const std::vector> & feat_set, const std::vector,std::allocator>> & gpair, xgboost::DMatrix * p_fmat) Line 474 C++
xgboost.dll!xgboost::tree::ColMaker::Builder::FindSplit(int depth, const std::vector> & qexpand, const std::vector,std::allocator>> & gpair, xgboost::DMatrix * p_fmat, xgboost::RegTree * p_tree) Line 482 C++
xgboost.dll!xgboost::tree::ColMaker::Builder::Update(const std::vector,std::allocator>> & gpair, xgboost::DMatrix * p_fmat, xgboost::RegTree * p_tree) Line 177 C++
xgboost.dll!xgboost::tree::ColMaker::Update(xgboost::HostDeviceVector> * gpair, xgboost::DMatrix * dmat, const std::vector> & trees) Line 116 C++
xgboost.dll!xgboost::gbm::GBTree::BoostNewTrees(xgboost::HostDeviceVector> * gpair, xgboost::DMatrix * p_fmat, int bst_group, std::vector>,std::allocator>>> * ret) Line 301 C++
xgboost.dll!xgboost::gbm::GBTree::DoBoost(xgboost::DMatrix * p_fmat, xgboost::HostDeviceVector> * in_gpair, xgboost::PredictionCacheEntry * predt) Line 196 C++
xgboost.dll!xgboost::LearnerImpl::UpdateOneIter(int iter, std::shared_ptr train) Line 977 C++
xgboost.dll!XGBoosterUpdateOneIter(void * handle, int iter, void * dtrain) Line 441 C++
```
The code is compiled with a few versions of MSVS2019, the latest is 16.7.2 used for compilation of XGBoost 1.2.0 . CUDA was enabled during the compilation, but the CPU exact method is used in this experiment.
Sorry that I can't provide the program used for training for several reasons, e.g. it's large and it requires external data to train on. Below is what I can provide instead:
- The training data size is 1137421 samples, each 4323 features (and I have 192GB RAM on the machine, so it doesn't look like an out of memory problem)
- The validation data contains 139171 samples.
- The error happens occasionally depending on the randomization
- I also have a minidump file, it's ~8GB archived, I can upload it if you point me where to.
- The exact revision is `0cd0dad0b5723112f9971659778c9f8b922c16f2` from branch `release_1.2.0`
Below you can find the parameters I used for XGBoost:
```
safe_xgboost(XGBoosterSetParam(h_booster, "tree_method", "exact"));
safe_xgboost(XGBoosterSetParam(h_booster, "colsample_bytree", "0.5"));
safe_xgboost(XGBoosterSetParam(h_booster, "colsample_bylevel", "0.5"));
safe_xgboost(XGBoosterSetParam(h_booster, "colsample_bynode", "0.5"));
safe_xgboost(XGBoosterSetParam(h_booster, "gamma", "100000"));
safe_xgboost(XGBoosterSetParam(h_booster, "reg_alpha", "10000"));
safe_xgboost(XGBoosterSetParam(h_booster, "reg_lambda", "10000"));
safe_xgboost(XGBoosterSetParam(h_booster, "learning_rate", "0.1"));
safe_xgboost(XGBoosterSetParam(h_booster, "max_depth", "11"));
```
Please, let me know how else I can help you with troubleshooting the issue and thank you in advance for looking into it.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.