aws / aws/amazon-sagemaker-examples

[Bug Report]

Open
#3,783 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Describe the bug**
When running a sagemaker batch transform job, the data, which is being ingested from S3 and is stored as a CSV is not loading the data right. For example The CSV is formatted as such

```
"joe annes rifle accesories discount"

"cute puppies for sale"

"Two dudes talk about sports"

"Smith & Wesson M&P 500 review"

"Glock vs 1911 handgun"
```
When I run this code instead of the batch job taking the CSV and breaking up the examples with every space, which is what

split_type="Line"

is telling the algorithm to do, but instead it just ingests all of the sentences in the above CSV, and outputs 1 probability.

When I print the input payload it looks like this

```
"joe annes rifle accesories discount"

2022-10-26T21:03:04,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -

"cute puppies for sale"

2022-10-26T21:03:04,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -

"Two dudes talk about sports"

2022-10-26T21:03:04,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle
```
Where each sentence is an inference example, and is separated by this statement

2022-10-26T21:03:04,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle

If I print the `request_body` in` input_fn(request_body, request_content_type)` It prints out all of the individual characters, which basically means sagemaker seems to be concatenating all of the sentences as such
`"joe annes rifle accesories discountcute puppies for saleTwo dudes talk about sports"`

Note that I can get this to work by running `splitlines()` on the `request_body` parameter. But this seems hacky, and will break if I want to merge non inference data at the end of the batch transform

**To reproduce**
put this code in a sagemaker notebook

```
import sagemaker
from sagemaker import get_execution_role
from sagemaker.utils import name_from_base
from sagemaker.pytorch import PyTorchModel
from sagemaker.predictor import RealTimePredictor, json_serializer, json_deserializer
import boto3
import json
import pprint
import time
from sagemaker.huggingface.model import HuggingFaceModel
role = sagemaker.get_execution_role()
```

```
elec_model = HuggingFaceModel(model_data='s3://company-models/multi-label-models.tar.gz',
role=role,
entry_point='torchserve_batch',
source_dir='source_dir',
transformers_version="4.17.0",
pytorch_version='1.10.2',
py_version='py38')
```

```
nl_detector = elec_model.transformer(
instance_count = 1,
instance_type = 'ml.p3.2xlarge', accept = "text/csv", strategy="MultiRecord", assemble_with="Line", output_path = "s3://company-batch-preds/ex_rob")#, max_payload = 10)
```
```
nl_detector.transform("s3://company-stage/test-run-30-million/", content_type="text/csv", split_type="Line", input_filter='$[2]', join_source='Input', output_filter='$[0, -1]')`
```

The in a directory source_dir/torchserve_batch.py, run

```
import numpy as np
import os
import sys
import logging
import json
import math

import torch
from sagemaker_inference import content_types, decoder

#import subprocess
#subprocess.check_call([sys.executable, "-m", "pip", "install", 'transformers'])

from transformers import ElectraTokenizer

def model_fn(model_dir):
"""
Load the model for inference
"""
print("running model")
model= torch.load(model_dir + "/model.pth")

model.eval()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

#use_cuda = torch.cuda.is_available()
#if use_cuda:
# model.cuda()
##return model.to(device)
return model


def predict_fn(input_data, model):
"""
Apply model to the incoming request
"""
tokenizer = ElectraTokenizer.from_pretrained('google/electra-base-discriminator')
sm = torch.nn.Softmax(dim=2)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
encoded_inputs = tokenizer(input_data, max_length=220, padding = 'longest', truncation = True)
tokened_words = encoded_inputs['input_ids']
attention_mask = encoded_inputs['attention_mask']

probs_li = []
first_indice = 0
second_indice = 400
add_to_indice = 400
iterations = math.ceil(len(input_data) / add_to_indice)
with torch.no_grad():
for _ in range(iterations):

outputs = model(torch.tensor(tokened_words[first_indice:second_indice]).to(device), torch.tensor(attention_mask[first_indice:second_indice]).to(device))
probs = torch.nn.functional.softmax(outputs.logits, 1)
probs_li.extend(probs.tolist())
first_indice += add_to_indice
second_indice += add_to_indice
#outputs = model(torch.tensor(tokened_words).to(device), torch.tensor(attention_mask).to(device))
#probs = torch.nn.functional.softmax(outputs.logits, 1)

return probs_li


def input_fn(request_body, request_content_type):
"""
Deserialize and prepare the prediction input
"""
print("listy", list(request_body))
print('requ_body', request_body)
print('request_split', request_body.splitlines())
#print('requ_body_split_lines', request_body.splitlines())
return request_body.splitlines()
#return request_body
#if request_content_type == "application/json":
# request = json.loads(request_body)
# return request


def output_fn(prediction_output, accept):
print("prediction_output", prediction_output)
print('this thing', accept)
return prediction_output#.tolist()
```

Contributor guide

Open the contributing guide

Research direction

Start with the notebook reproduction and source_dir/torchserve_batch.py, especially input_fn(request_body, request_content_type) and predict_fn. Run the SageMaker batch transform example with the supplied CSV and verify that each CSV line reaches inference as a separate example with matching probabilities, including when joined input data is used.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.