aws / aws/amazon-sagemaker-examples
[Bug Report]
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Describe the bug**
When running a sagemaker batch transform job, the data, which is being ingested from S3 and is stored as a CSV is not loading the data right. For example The CSV is formatted as such
```
"joe annes rifle accesories discount"
"cute puppies for sale"
"Two dudes talk about sports"
"Smith & Wesson M&P 500 review"
"Glock vs 1911 handgun"
```
When I run this code instead of the batch job taking the CSV and breaking up the examples with every space, which is what
split_type="Line"
is telling the algorithm to do, but instead it just ingests all of the sentences in the above CSV, and outputs 1 probability.
When I print the input payload it looks like this
```
"joe annes rifle accesories discount"
2022-10-26T21:03:04,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
"cute puppies for sale"
2022-10-26T21:03:04,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
"Two dudes talk about sports"
2022-10-26T21:03:04,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle
```
Where each sentence is an inference example, and is separated by this statement
2022-10-26T21:03:04,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle
If I print the `request_body` in` input_fn(request_body, request_content_type)` It prints out all of the individual characters, which basically means sagemaker seems to be concatenating all of the sentences as such
`"joe annes rifle accesories discountcute puppies for saleTwo dudes talk about sports"`
Note that I can get this to work by running `splitlines()` on the `request_body` parameter. But this seems hacky, and will break if I want to merge non inference data at the end of the batch transform
**To reproduce**
put this code in a sagemaker notebook
```
import sagemaker
from sagemaker import get_execution_role
from sagemaker.utils import name_from_base
from sagemaker.pytorch import PyTorchModel
from sagemaker.predictor import RealTimePredictor, json_serializer, json_deserializer
import boto3
import json
import pprint
import time
from sagemaker.huggingface.model import HuggingFaceModel
role = sagemaker.get_execution_role()
```
```
elec_model = HuggingFaceModel(model_data='s3://company-models/multi-label-models.tar.gz',
role=role,
entry_point='torchserve_batch',
source_dir='source_dir',
transformers_version="4.17.0",
pytorch_version='1.10.2',
py_version='py38')
```
```
nl_detector = elec_model.transformer(
instance_count = 1,
instance_type = 'ml.p3.2xlarge', accept = "text/csv", strategy="MultiRecord", assemble_with="Line", output_path = "s3://company-batch-preds/ex_rob")#, max_payload = 10)
```
```
nl_detector.transform("s3://company-stage/test-run-30-million/", content_type="text/csv", split_type="Line", input_filter='$[2]', join_source='Input', output_filter='$[0, -1]')`
```
The in a directory source_dir/torchserve_batch.py, run
```
import numpy as np
import os
import sys
import logging
import json
import math
import torch
from sagemaker_inference import content_types, decoder
#import subprocess
#subprocess.check_call([sys.executable, "-m", "pip", "install", 'transformers'])
from transformers import ElectraTokenizer
def model_fn(model_dir):
"""
Load the model for inference
"""
print("running model")
model= torch.load(model_dir + "/model.pth")
model.eval()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
#use_cuda = torch.cuda.is_available()
#if use_cuda:
# model.cuda()
##return model.to(device)
return model
def predict_fn(input_data, model):
"""
Apply model to the incoming request
"""
tokenizer = ElectraTokenizer.from_pretrained('google/electra-base-discriminator')
sm = torch.nn.Softmax(dim=2)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
encoded_inputs = tokenizer(input_data, max_length=220, padding = 'longest', truncation = True)
tokened_words = encoded_inputs['input_ids']
attention_mask = encoded_inputs['attention_mask']
probs_li = []
first_indice = 0
second_indice = 400
add_to_indice = 400
iterations = math.ceil(len(input_data) / add_to_indice)
with torch.no_grad():
for _ in range(iterations):
outputs = model(torch.tensor(tokened_words[first_indice:second_indice]).to(device), torch.tensor(attention_mask[first_indice:second_indice]).to(device))
probs = torch.nn.functional.softmax(outputs.logits, 1)
probs_li.extend(probs.tolist())
first_indice += add_to_indice
second_indice += add_to_indice
#outputs = model(torch.tensor(tokened_words).to(device), torch.tensor(attention_mask).to(device))
#probs = torch.nn.functional.softmax(outputs.logits, 1)
return probs_li
def input_fn(request_body, request_content_type):
"""
Deserialize and prepare the prediction input
"""
print("listy", list(request_body))
print('requ_body', request_body)
print('request_split', request_body.splitlines())
#print('requ_body_split_lines', request_body.splitlines())
return request_body.splitlines()
#return request_body
#if request_content_type == "application/json":
# request = json.loads(request_body)
# return request
def output_fn(prediction_output, accept):
print("prediction_output", prediction_output)
print('this thing', accept)
return prediction_output#.tolist()
```
Contributor guide
Research direction
Start with the notebook reproduction and source_dir/torchserve_batch.py, especially input_fn(request_body, request_content_type) and predict_fn. Run the SageMaker batch transform example with the supplied CSV and verify that each CSV line reaches inference as a separate example with matching probabilities, including when joined input data is used.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python, pytorch
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100