dotnet / dotnet/machinelearning
image classification api - cudnn fail to initialize when using GPU to train
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
### System information
GPU: GTX 1060
TF binary: 2.3.1
Cuda: 10.1
Cudnn: 7.6.4
### Issue
Error log
>Pipeline crashed: xf=ValueToKeyMapping{ col=Label:Label} xf=RawByteImageLoading{ col=ImageSource_featurized:ImageSource imageFolder=} xf=ColumnCopying{ col=Features:ImageSource_featurized} tr=ImageClassification{} xf=KeyToValueMapping{ col=PredictedLabel:PredictedLabel} cache=- . Exception: Tensorflow.TensorflowException: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node resnet_v2_50/conv1/Conv2D}}]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node resnet_v2_50/conv1/Conv2D}}]]
[[resnet_v2_50/SpatialSqueeze/_7]]
0 successful operations.
0 derived errors ignored.
at Microsoft.ML.TensorFlow.TensorFlowUtils.Runner.Run()
at Microsoft.ML.Vision.ImageClassificationTrainer.CacheFeaturizedImagesToDisk(IDataView input, String labelColumnName, String imageColumnName, ImageProcessor imageProcessor, String inputTensorName, String outputTensorName, String cacheFilePath, Dataset dataset, Action`1 metricsCallback, Nullable`1 validationFraction)
at Microsoft.ML.Vision.ImageClassificationTrainer.TrainModelCore(TrainContext trainContext)
at Microsoft.ML.Trainers.TrainerEstimatorBase`2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor)
at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
at Microsoft.ML.AutoML.RunnerUtil.TrainAndScorePipeline[TMetrics](MLContext context, SuggestedPipeline pipeline, IDataView trainData, IDataView validData, String groupId, String labelColumn, IMetricsAgent`1 metricsAgent, ITransformer preprocessorTransform, FileInfo modelFileInfo, DataViewSchema modelInputSchema, IChannel logger) (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2021-04-07 22:33:17.0887 TRACE [Source=AutoML, Kind=Error] Pipeline crashed: xf=ValueToKeyMapping{ col=Label:Label} xf=RawByteImageLoading{ col=ImageSource_featurized:ImageSource imageFolder=} xf=ColumnCopying{ col=Features:ImageSource_featurized} tr=ImageClassification{} xf=KeyToValueMapping{ col=PredictedLabel:PredictedLabel} cache=- . Exception: Tensorflow.TensorflowException: 2 root error(s) found.
### Suggest fix
After some investigation, it appears to be a bug in TF 2.x ( see this [issue](https://github.com/tensorflow/tensorflow/issues/44885). And after I set `TF_FORCE_GPU_ALLOW_GROWTH` to true, the error has gone.
However, a better place to fix this bug might be in ImageClassification API, where it can call
>tf.config.experimental.set_memory_growth(physical_devices[0], True)
before loading model so that user doesn't need to set up envrionment variable everytime they uses this API
Contributor guide
Assessment
This issue has not been assessed yet.