aws / aws/amazon-sagemaker-examples

[Example Request]JumpStart manifest file usage for object classification

Open
#3,895 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Describe the use case example you want to see**
An example of how you would use a manifest file instead of pointing at the images directories for the following example notebook.
https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart_image_classification/Amazon_JumpStart_Image_Classification.ipynb

**How would this example be used? Please describe.**
With a Ground Truth manifest file.

**Describe which SageMaker services are involved**
SageMaker JumpStart

**Describe what other services (other than SageMaker) are involved***
Ground Truth

**Describe which dataset could be used. Provide its location in s3://sagemaker-sample-files or another source.**
I am trying to create a TrainingInput that will work with the example notebook provided, but I keep getting errors saying that there are no files.

I have tried both ManifestFile and AugmentedManifest file. Here is an example of what I have tried:

ti = TrainingInput(s3_data="s3://some-bucket/auto-label.manifest",
distribution='FullyReplicated',
s3_data_type="AugmentedManifestFile",
input_mode="Pipe", #"File for ManifestFile
attribute_names=["source-ref", "auto-label-metadata"],
)
ic_estimator.fit(inputs={"train": ti}, logs=True)

And here is what the .manifest file looks like:

{"source-ref": "s3://image-bucket/sig_extracted/my-folder/run2/some-img.png", "auto-label-metadata": {"confidence": 1, "job-name": "labeling-job/auto-label", "class-name": "class1", "human-annotated": "yes", "creation-date": "2023-04-03T15:11:14.901287", "type": "groundtruth/image-classification-multilabel"}}
{"source-ref": "s3://.... cont'd

Error:
Images for this model will be resized to image size: [299, 299]
Traceback (most recent call last):
File "/opt/ml/code/transfer_learning.py", line 89, in
trainer.train()
File "/usr/local/lib/python3.9/site-packages/sagemaker_jumpstart_tensorflow_script_utilities/trainer.py", line 75, in train
datasets = self._create_datasets_check_type(model_info)
File "/usr/local/lib/python3.9/site-packages/sagemaker_jumpstart_tensorflow_script_utilities/trainer.py", line 124, in _create_datasets_check_type
datasets = self.task.datasets_class.from_config(
File "/opt/ml/code/datasets.py", line 73, in from_config
dataset_train = image_dataset_from_directory(str(train_channel), **dataset_train_kwargs)
File "/usr/local/lib/python3.9/site-packages/keras/utils/image_dataset.py", line 192, in image_dataset_from_directory
image_paths, labels, class_names = dataset_utils.index_directory(
File "/usr/local/lib/python3.9/site-packages/keras/utils/dataset_utils.py", line 66, in index_directory
for subdir in sorted(tf.io.gfile.listdir(directory)):
File "/usr/local/lib/python3.9/site-packages/tensorflow/python/lib/io/file_io.py", line 766, in list_directory_v2
raise errors.NotFoundError(
tensorflow.python.framework.errors_impl.NotFoundError: Could not find directory None

Contributor guide

Open the contributing guide

Research direction

Start with introduction_to_amazon_algorithms/jumpstart_image_classification/Amazon_JumpStart_Image_Classification.ipynb and review how TrainingInput is configured for image directories. Run the notebook with the Ground Truth manifest details from the issue and trace the reported None directory error. Done means the notebook demonstrates manifest-file usage for object classification without the missing-directory failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, keras, python, tensorflow
Domain
cloud, data, machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.