[GSProcessing] Put Entry Script inside the Docker to avoid version mismatch
- Dominant language
- Python
- Stars
- 452
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
Issue we have observed because distributed_executor.py contains lots of code and sometimes includes breaking changes between versions:
* The entry point is uploaded and attached during job launch/pipeline creation, from local graphstorm repo
* If a user cloned the graphstorm repo from main, they get the latest version of the entry point, which in might contain a backwards incompatible change
* When launching a sagemaker job, it will use the latest entry point, on a container that was built using e.g v0.4.2
* As a result, it's looking for library function that was renamed, causing an error.
Ideally the entry point should be very thin, containing argument parsing and one library function call which doesn't change
Any backwards-incompatible changes to the entry point need to marked as breaking (e.g. changing the name of one of its imports)
Contributor guide
Research direction
Start by tracing how distributed_executor.py is uploaded during job launch or pipeline creation and how the SageMaker container uses it. Review the Docker/container version boundary and define the stable library call the entry script should use. Done means the entry point no longer risks mismatching the container version, with breaking entry-point changes explicitly marked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python
- Domain
- cloud, distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100