aws / aws/amazon-sagemaker-examples

[Bug Report] "Complete with labeling errors" and wrong labels in the output

Open
#2,928 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
Add the link to the notebook. I do not have a notebook but my settings file is available [here](https://drive.google.com/file/d/1IQBMUQIAclx3K9wjufcGtIk7GzItvMuw/view).

**Describe the bug**
I came up on 2 issues when using SageMaker for an image labeling task that had 6252 images to label using Private Workforce - I assigned them to myself as a first test.

My job status was "Complete with labeling errors" with only 5853 / 6252 = Labeled / total dataset objects
I labeled the objects over 4 days and thus 3 expired at the end of each day so I expected to have 3 dataset objects missing but not 399. What could the reason for this be?

When looking at the output labels quite a few (hundreds) of them are clearly wrong - given that I labeled them all myself this is impossible (I may have missed 1 or 2 but not hundreds) so could this be related to the issue above. What else could have gone wrong with this?

Thanks so much!!

**To reproduce**
A clear, step-by-step set of instructions to reproduce the bug.

**Logs**
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.

Contributor guide

Open the contributing guide

Research direction

Start with the linked settings file and the reported SageMaker Private Workforce labeling task. Reproduce the 6252-object run, compare the 5853 labeled objects with the expected expirations, and inspect the resulting output labels and available logs; done means identifying a reproducible cause for both the missing objects and incorrect labels.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.