awslabs / awslabs/sagemaker-debugger
Provide a way to randomly access a step
- Dominant language
- Python
- Stars
- 165
- Forks
- 82
- PR merge metrics
- No merged PRs in 30d
Description
Current impl :
If there are n steps present, all n steps index would be downloaded before the call finishes.
We should allow a way to randomly access step.
Use case:
1) I want the tensor from last step. or I want the tensor from step 5000-8000
In both these cases, user is only interested in certain steps rather than all steps while doing analysis.
Estimation of time -
If there are 50K saved , it will take 50 list calls + 50K get calls( which should be in ~ 20 sec according to benchmark https://github.com/awslabs/sagemaker-debugger/pull/80 where 10K files take 4 sec) to get the index file. 50 list calls can be expensive as those are sequential operation and list is expensive operation.
Contributor guide
Research direction
Start by tracing the current tensor-step indexing and download flow described in the issue, then review benchmark pull request #80 for the existing list and get-call behavior. Done should mean users can access a selected step or step range without downloading every saved step first.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100