microsoft / microsoft/SynapseML
TuneHyperparameters - Exception thrown in awaitResult
@imatiach-msft is already working on this.
Since Aug 26, 2019.
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
[log4j.txt](https://github.com/Azure/mmlspark/files/3539284/log4j.txt)
Following this example: https://github.com/Azure/mmlspark/blob/master/notebooks/samples/HyperParameterTuning%20-%20Fighting%20Breast%20Cancer.ipynb
```
from mmlspark.automl import TuneHyperparameters
from mmlspark.train import TrainClassifier
from pyspark.ml.classification import LogisticRegression, RandomForestClassifier, GBTClassifier
logReg = LogisticRegression()
randForest = RandomForestClassifier()
smlmodels = [logReg, randForest]
from mmlspark.automl import *
paramBuilder = \
HyperparamBuilder() \
.addHyperparam(logReg, logReg.regParam, RangeHyperParam(0.1, 0.3)) \
.addHyperparam(randForest, randForest.numTrees, DiscreteHyperParam([5,10])) \
searchSpace = paramBuilder.build()
# The search space is a list of params to tuples of estimator and hyperparam
print(searchSpace)
randomSpace = RandomSpace(searchSpace)
bestModel = TuneHyperparameters(
evaluationMetric="accuracy", models=smlmodels, numFolds=1,
numRuns=len(smlmodels) * 1, parallelism=1,
paramSpace=randomSpace.space(), seed=0).fit(data.select("features", "label"))
```
Setting numFolds etc. larger simply increases the time to throw an exception.
The data is in a PySpark style format. A huge feature vector + a ValueIndexed label. Hence I do not call TrainClassifier on the models. I did attempt to use TrainClassifier (which can train a single model) but it still throws the same error.
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 22 in stage 36.0 failed 4 times, most recent failure: Lost task 22.3 in stage 36.0 (TID 843, 10.179.68.7, executor 2): java.util.NoSuchElementException: key not found: 85
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.