Lightning-AI / Lightning-AI/pytorch-lightning

Add support S3 as a storage option for profiling results

Open
#20,348 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature profiler
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

Description: Currently, when using the default_root_dir parameter to specify an S3 bucket as the storage location for profiling results, the profiler fails to work due to lack of support for S3 I/O. This limitation prevents users from storing profiling results in a scalable and durable storage solution.

Motivation: As a user of the Lightning library, I would like to be able to store profiling results in an S3 bucket to take advantage of its scalability, durability, and ease of access. This would enable me to easily collect and analyze profiling data from multiple runs, and share the results with others.

### Pitch

I propose that the Lightning library be extended to support S3 as a storage option for profiling results. This could be achieved by adding S3 I/O support to the profiler, allowing users to specify an S3 bucket as the default_root_dir. This would enable users to store profiling results in a flexible and scalable storage solution, and would greatly enhance the usability and usefulness of the profiling feature.

### Alternatives

_No response_

### Additional context

_No response_

cc @lantiga @borda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how the profiler uses default_root_dir and where profiling results are written, then identify the existing storage I/O boundary. Define the S3 read and write behavior needed for profiling results and verify it with coverage for an S3-backed default_root_dir.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.