Lightning-AI / Lightning-AI/pytorch-lightning
Add support S3 as a storage option for profiling results
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
Description: Currently, when using the default_root_dir parameter to specify an S3 bucket as the storage location for profiling results, the profiler fails to work due to lack of support for S3 I/O. This limitation prevents users from storing profiling results in a scalable and durable storage solution.
Motivation: As a user of the Lightning library, I would like to be able to store profiling results in an S3 bucket to take advantage of its scalability, durability, and ease of access. This would enable me to easily collect and analyze profiling data from multiple runs, and share the results with others.
### Pitch
I propose that the Lightning library be extended to support S3 as a storage option for profiling results. This could be achieved by adding S3 I/O support to the profiler, allowing users to specify an S3 bucket as the default_root_dir. This would enable users to store profiling results in a flexible and scalable storage solution, and would greatly enhance the usability and usefulness of the profiling feature.
### Alternatives
_No response_
### Additional context
_No response_
cc @lantiga @borda
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the profiler uses default_root_dir and where profiling results are written, then identify the existing storage I/O boundary. Define the S3 read and write behavior needed for profiling results and verify it with coverage for an S3-backed default_root_dir.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100