awslabs / awslabs/graphstorm

If a GSProcessing job fails at the re-partition step follow-up jobs fail without clear reason why

Open
#1,238 0 comments 0 reactions 1 assignee Claimed by @thvasilo View on GitHub
0.5.1 dist-partition gsprocessing
Dominant language
Python
Stars
452
Forks
76
PR merge metrics
No merged PRs in 30d

Description

When users set `--do-repartition True` for a GSProcessing job, there's a chance that job will fail.

Within GSP we don't fail the entire job when that happens, in order to not waste all the GSProcessing work that's been done before, but rather log an error/warning.

Without looking at the logs, an end user cannot know that they will need to run the follow-up re-partition job independently.

If they try to run the DistPart job, they will could get an error about a reshape operation failing, without a clear reason why.

We should provide persistent indicators for failed re-partition jobs that

1. Allow users to know something is wrong when looking at the file output
2. Allow us to fail the DistPart job early with a more descriptive error message.

Also we should include checks in the beginning of the DistPart job that checks for the expected shape of label/mask vectors and provide descriptive errors.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.