alibaba / alibaba/GraphScope

flex-coordinator proposal

Open
#3,191 0 comments 0 reactions 1 assignee Claimed by @lidongze0629 View on GitHub
component:coordinator
Dominant language
C++
Stars
3.6k
Forks
468
Avg merge
29m
Merged PRs (30d)
1

Description

**Case 1: GAE for offline graph analytics**
----

**Impl GraphScope operator**

**1. How to submit jobs**

Users submit a CRD resource as a batch job by `kubectl`, and each job will be run in a new set of pods. For custom algos, users need to package them in an image and run the container as a sidecar.

**2. Coordinator responsibility**

The coordinator is responsible for 1) identifying the engine pods that will be used in running the MPI job; 2) compiling and distributing the algorithms to the engine machines if needed; 3) managing the execution of MPI job on the engine machines.

**3. Monitor**

![dashboard-ui](https://github.com/alibaba/GraphScope/assets/23491543/65423c56-8c1a-478d-90cf-7d580664d73a)

We may use K8s Dashboard to monitor the GraphScope operator status, including:
- monitor and record the job, including the running jobs, failed jobs
- cpu/memory of each pod
- pod log (GAE running log)

But I'm not sure it can store the logs of pods that have completed their execution.

**4. CRD**
```yaml
apiVersion: graphscope.io/v1beta1
kind: AnalyticalJob
metadata:
name: x
spec:
algorithm:
custom_algorithm: true
custom_algorithm_image: "myalgo:latest"
name: sssp
runtime_parameters:
- key: source
value: 1
input:
oid_type: "int64_t"
vertex_map: "global"
vertices:
- label: student
loader:
source: "s3://test/student.csv"
delimiter: ","
header_row: true
filetype: "CSV"
args:
edges:
- label: teacher_student
loader:
source: "s3://test/teacher_student.csv"
delimiter: ","
header_row: true
filetype: "CSV"
args:
output:
location: "s3://test/result"
Worker:
replicas: 2
template:
spec:
containers:
- image: graphscope/analytical:latest
name: engine
resources:
limits:
cpu: 2
memory: 4Gi
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.