awslabs / awslabs/sagemaker-debugger

Provide a way to randomly access a step

Open
#99 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
165
Forks
82
PR merge metrics
No merged PRs in 30d

Description

Current impl :
If there are n steps present, all n steps index would be downloaded before the call finishes.

We should allow a way to randomly access step.
Use case:
1) I want the tensor from last step. or I want the tensor from step 5000-8000

In both these cases, user is only interested in certain steps rather than all steps while doing analysis.

Estimation of time -
If there are 50K saved , it will take 50 list calls + 50K get calls( which should be in ~ 20 sec according to benchmark https://github.com/awslabs/sagemaker-debugger/pull/80 where 10K files take 4 sec) to get the index file. 50 list calls can be expensive as those are sequential operation and list is expensive operation.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the current tensor-step indexing and download flow described in the issue, then review benchmark pull request #80 for the existing list and get-call behavior. Done should mean users can access a selected step or step range without downloading every saved step first.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.