[GSPartition] Failure running on AWS Batch for Parmetis
- Dominant language
- Python
- Stars
- 452
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
When we are using parmetis, we see failure when running on aws batch.
Like we are now running with ``mpirun -np {num_parts}``, we should make num_parts an input argument.
Previously during 30 partitions, we have errors:
```
[Rank: 30] Reading file: /mount/test-graph-folder/demo-graph/edge-type1/edge-file.csv
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 3 with PID 0 on node ip-172-31-71-140 exited on signal 4 (Illegal instruction).
--------------------------------------------------------------------------
```
The error can be resolved if we change ``mpirun -np {num_parts}`` to ``mpirun -np 1``.
However when we are running dataset with large partition numbers (like >=16) it may failed as all the processes are referring to the same file at the same time. It is better to adjust it to an input.
Contributor guide
Research direction
No file or test is named. Start by locating the GSPartition code that assembles the `mpirun -np {num_parts}` command and inspect how it runs in AWS Batch; make the process count an input and verify behavior for one process and larger partition counts such as 16 or more.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100