awslabs / awslabs/graphstorm

[GSPartition] Failure running on AWS Batch for Parmetis

Open
#1,218 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
452
Forks
76
PR merge metrics
No merged PRs in 30d

Description

When we are using parmetis, we see failure when running on aws batch.

Like we are now running with ``mpirun -np {num_parts}``, we should make num_parts an input argument.

Previously during 30 partitions, we have errors:
```
[Rank: 30] Reading file: /mount/test-graph-folder/demo-graph/edge-type1/edge-file.csv
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 3 with PID 0 on node ip-172-31-71-140 exited on signal 4 (Illegal instruction).
--------------------------------------------------------------------------
```

The error can be resolved if we change ``mpirun -np {num_parts}`` to ``mpirun -np 1``.

However when we are running dataset with large partition numbers (like >=16) it may failed as all the processes are referring to the same file at the same time. It is better to adjust it to an input.

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by locating the GSPartition code that assembles the `mpirun -np {num_parts}` command and inspect how it runs in AWS Batch; make the process count an input and verify behavior for one process and larger partition counts such as 16 or more.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, distributed-systems
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.