Malloc error converting sklearn decision tree
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
I've trained a model using scikit-learn's DecisionTreeClassifier on a dataset with 1,600,000 rows, 15 features, max_depth=7. When I try to convert using coremltools for a model over about 2 mb, I get the following error:
`malloc: *** error for object 0x7fb096a0f738: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug`
I might be doing this part wrong, but when I try to log and debug, I get:
`Segmentation fault: 11`
To recreate:
```python
import pandas as pd
import numpy as np
import sklearn
import coremltools
import random
import string
from sklearn.tree import DecisionTreeClassifier
X = np.random.choice([0, 1], size=(15*1000,), p=[1./3, 2./3])
X = np.split(X,1000)
y = []
for i in range(0,1000):
x = ''.join(random.choice(string.lowercase) for x in range(5))
y.append(x)
clf = DecisionTreeClassifier()
clf.fit(X,y)
d = {'arr': X, 'str': y}
df = pd.DataFrame(data=d)
coreml_model = coremltools.converters.sklearn.convert(clf, "arr",
"str") ## malloc error
coreml_model.save('mymodel.mlmodel')
```
Contributor guide
Research direction
Start with the provided Python reproduction and the coremltools.converters.sklearn.convert entry point, comparing the reported large model with the smaller example. Reproduce the malloc error or segmentation fault and identify a conversion outcome that completes without either failure for the DecisionTreeClassifier.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100