New RestServer Architecture: RestServer -> DB -> ApiServer
Open
@hzy46 is already working on this.
Since Jun 28, 2020.
pai-dev
- Dominant language
- JavaScript
- Stars
- 2.7k
- Forks
- 554
- Avg merge
- 20h 42m
- Merged PRs (30d)
- 14
Description
By leveraging DB, RestServer can be
- RAW Consistency
- High Perf and Powerful Query: List, Paging, Sorting, Summarizing, etc
- Larger storage quota and duration
- Active and history jobs are unified and merged together naturally (no need to introduce UID) (https://github.com/microsoft/pai/issues/3935)
- Job Name can submit idempotently, attach metadata arbitrarily, and query uniquely (https://github.com/microsoft/pai/issues/3935)
- etc
Features depend on it
List History Jobs: https://github.com/microsoft/pai/issues/3845, https://github.com/microsoft/pai/issues/4610, https://github.com/microsoft/pai/issues/3935
Expose K8s events: part of enrich job debugging info: https://github.com/microsoft/pai/issues/4649
New RestServer Architecture
In short, compared with current architecture, we insert a DB between RestServer and ApiServer.

Sub Tasks
- Database Controller (StatefulSet with initializer, write merger, watcher, poller, gcer)
- P0 Database schema
- P0 Database ORM interface in Node.js
- P0 Service setup (sevice.yaml, start.sh, config, ... , etc)
- Initializer (handle schema sync, upgrade, legacy, etc...)
- P0 Version table setup
- P1 Legacy framework transfer @suiguoxin
- P0 Write merger
- P0.5 DB poller
- API watcher
- P0 Watcher for framework
- P2 Watcher for event
- P2 Watcher for pod
- #5653
- Rest API Change
- P0 POST /api/v2/jobs
- P0 GET /api/v2/jobs
- P0 GET /api/v2/jobs/{user}~{job}
- P0 GET /api/v2/jobs/{user}~{job}/config
- P0 PUT /api/v2/jobs/{user}~{job}/executionType
- P0 Swagger update
- P1 /api/v2/jobs/{user}~{job}/job-attempts @suiguoxin #4716
- P1 /api/v2/jobs/{user}~{job}/job-attempts/{attemptIndex} @suiguoxin
- Webportal
- P1 paging on list job page
- Fluentd @suiguoxin #4716
- P1 Fluentd plugin multithread
- P1 Fluentd change for new schema (for framework history)
- P2 Fluentd for Pod
- Other
- P1 Framework controller Decrease GC time?
- P2 Make postgresql a StatefulSet
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.