microsoft / microsoft/SynapseML

TuneHyperparameters - Exception thrown in awaitResult

Open
#667 4 comments 0 reactions 1 assignee View on GitHub

@imatiach-msft is already working on this.

Since Aug 26, 2019.

bug
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

[log4j.txt](https://github.com/Azure/mmlspark/files/3539284/log4j.txt)

Following this example: https://github.com/Azure/mmlspark/blob/master/notebooks/samples/HyperParameterTuning%20-%20Fighting%20Breast%20Cancer.ipynb

```
from mmlspark.automl import TuneHyperparameters
from mmlspark.train import TrainClassifier
from pyspark.ml.classification import LogisticRegression, RandomForestClassifier, GBTClassifier
logReg = LogisticRegression()
randForest = RandomForestClassifier()
smlmodels = [logReg, randForest]

from mmlspark.automl import *

paramBuilder = \
HyperparamBuilder() \
.addHyperparam(logReg, logReg.regParam, RangeHyperParam(0.1, 0.3)) \
.addHyperparam(randForest, randForest.numTrees, DiscreteHyperParam([5,10])) \

searchSpace = paramBuilder.build()
# The search space is a list of params to tuples of estimator and hyperparam
print(searchSpace)
randomSpace = RandomSpace(searchSpace)

bestModel = TuneHyperparameters(
evaluationMetric="accuracy", models=smlmodels, numFolds=1,
numRuns=len(smlmodels) * 1, parallelism=1,
paramSpace=randomSpace.space(), seed=0).fit(data.select("features", "label"))
```

Setting numFolds etc. larger simply increases the time to throw an exception.

The data is in a PySpark style format. A huge feature vector + a ValueIndexed label. Hence I do not call TrainClassifier on the models. I did attempt to use TrainClassifier (which can train a single model) but it still throws the same error.

> org.apache.spark.SparkException: Job aborted due to stage failure: Task 22 in stage 36.0 failed 4 times, most recent failure: Lost task 22.3 in stage 36.0 (TID 843, 10.179.68.7, executor 2): java.util.NoSuchElementException: key not found: 85

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.