dotnet / dotnet/machinelearning

image classification api - cudnn fail to initialize when using GPU to train

Open
#5,739 2 comments 0 reactions 0 assignees View on GitHub
area-Vision Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

### System information

GPU: GTX 1060
TF binary: 2.3.1
Cuda: 10.1
Cudnn: 7.6.4

### Issue
Error log
>Pipeline crashed: xf=ValueToKeyMapping{ col=Label:Label} xf=RawByteImageLoading{ col=ImageSource_featurized:ImageSource imageFolder=} xf=ColumnCopying{ col=Features:ImageSource_featurized} tr=ImageClassification{} xf=KeyToValueMapping{ col=PredictedLabel:PredictedLabel} cache=- . Exception: Tensorflow.TensorflowException: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node resnet_v2_50/conv1/Conv2D}}]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node resnet_v2_50/conv1/Conv2D}}]]
[[resnet_v2_50/SpatialSqueeze/_7]]
0 successful operations.
0 derived errors ignored.
at Microsoft.ML.TensorFlow.TensorFlowUtils.Runner.Run()
at Microsoft.ML.Vision.ImageClassificationTrainer.CacheFeaturizedImagesToDisk(IDataView input, String labelColumnName, String imageColumnName, ImageProcessor imageProcessor, String inputTensorName, String outputTensorName, String cacheFilePath, Dataset dataset, Action`1 metricsCallback, Nullable`1 validationFraction)
at Microsoft.ML.Vision.ImageClassificationTrainer.TrainModelCore(TrainContext trainContext)
at Microsoft.ML.Trainers.TrainerEstimatorBase`2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor)
at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
at Microsoft.ML.AutoML.RunnerUtil.TrainAndScorePipeline[TMetrics](MLContext context, SuggestedPipeline pipeline, IDataView trainData, IDataView validData, String groupId, String labelColumn, IMetricsAgent`1 metricsAgent, ITransformer preprocessorTransform, FileInfo modelFileInfo, DataViewSchema modelInputSchema, IChannel logger) (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2021-04-07 22:33:17.0887 TRACE [Source=AutoML, Kind=Error] Pipeline crashed: xf=ValueToKeyMapping{ col=Label:Label} xf=RawByteImageLoading{ col=ImageSource_featurized:ImageSource imageFolder=} xf=ColumnCopying{ col=Features:ImageSource_featurized} tr=ImageClassification{} xf=KeyToValueMapping{ col=PredictedLabel:PredictedLabel} cache=- . Exception: Tensorflow.TensorflowException: 2 root error(s) found.

### Suggest fix
After some investigation, it appears to be a bug in TF 2.x ( see this [issue](https://github.com/tensorflow/tensorflow/issues/44885). And after I set `TF_FORCE_GPU_ALLOW_GROWTH` to true, the error has gone.

However, a better place to fix this bug might be in ImageClassification API, where it can call
>tf.config.experimental.set_memory_growth(physical_devices[0], True)

before loading model so that user doesn't need to set up envrionment variable everytime they uses this API

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.