Azure / Azure/azureml-examples

Error with score.py file : ResourceDeploymentFailure: ResourceNotReady: User container has crashed or terminated

Open
#1,977 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

### Operating System

Windows

### Version Information

I am using the .bin model which is a TensorFlow model and to predict it I need bertbaseuncased class too but score.py is returning this error - ResourceDeploymentFailure: ResourceNotReady: User container has crashed or terminated

**My score.py file code** :
import os
import numpy as np
import pandas as pd
import tensorflow as tf
from PIL import Image
from azureml.core import Model
import torch
import transformers
import torch.nn as nn
DEVICE="cpu" #cpu or gpu based on env

class BERTBaseUncased(nn.Module):
def __init__(self):
super(BERTBaseUncased, self).__init__()
self.bert = transformers.BertModel.from_pretrained("bert-base-uncased", return_dict=False) # num_labels=5
self.bert_drop = nn.Dropout(0.3)
self.out = nn.Linear(768, 5)

def forward(self, ids, mask, token_type_ids, return_dict=False):
_, o2 = self.bert(ids, attention_mask=mask, token_type_ids=token_type_ids, return_dict=return_dict)
bo = self.bert_drop(o2)
output = self.out(bo)
return output

def init():
global model

# AZUREML_MODEL_DIR is an environment variable created during deployment
# The path "model" is the name of the registered model's folder
model_path = os.path.join(os.environ["AZUREML_MODEL_DIR"], "model")

# load the model
model = BERTBaseUncased()
model.load_state_dict(torch.load(model_path, map_location=torch.device(DEVICE)))
model.to(DEVICE)
model.eval()

def run(mini_batch):
print(f"run method start: {__file__}, run({mini_batch})")
resultList = []
tokenizer = transformers.BertTokenizer.from_pretrained("bert-base-cased", do_lower_case=True)
max_len = 64

for inputs in mini_batch:
sp = str(inputs) #here we passed our description for prediction in flask, Not sure if mini batch has description
sp = " ".join(sp.split())
inputs = tokenizer.encode_plus(
sp,
None,
add_special_tokens=True,
max_length=max_len,
#pad_to_max_length=True,
)
ids = inputs["input_ids"]
mask = inputs["attention_mask"]
token_type_ids = inputs["token_type_ids"]

padding_length = max_len - len(ids)
ids = ids + ([0] * padding_length)
mask = mask + ([0] * padding_length)
token_type_ids = token_type_ids + ([0] * padding_length)

ids= torch.tensor(ids, dtype=torch.long).unsqueeze(0)
mask= torch.tensor(mask, dtype=torch.long).unsqueeze(0)
token_type_ids=torch.tensor(token_type_ids, dtype=torch.long).unsqueeze(0)
fin_outputs=[]

ids = ids.to(DEVICE)
token_type_ids = token_type_ids.to(DEVICE)
mask = mask.to(DEVICE)
outputs = model(ids=ids,mask=mask, token_type_ids=token_type_ids)
sft=torch.nn.Softmax(dim=1)(outputs)#.logits)
prediction=torch.argmax(sft, dim=1)
fin_outputs.extend(prediction.cpu().detach().numpy().tolist())
resultList.append([os.path.basename(inputs), fin_outputs])

return pd.DataFrame(resultList)

### Steps to reproduce

When trying uploading it on ml.azure.com

### Expected behavior

It should accept the score.py file and deployment should run further

### Actual behavior

It gives error in the score.py file : ResourceDeploymentFailure: ResourceNotReady: User container has crashed or terminated

### Addition information

Please check if I need to do changes in the score file

Contributor guide

Open the contributing guide

Research direction

Start by inspecting score.py, especially init() and run(), then review the Azure ML deployment logs for the container crash. Confirm the deployment can load the model and execute prediction requests without ResourceDeploymentFailure.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, numpy, pandas, python, pytorch, tensorflow
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.