adoptium / adoptium/aqa-test-tools

Increase the TRSS Server efficiency by running multiple node in parallel

Open
#108 0 comments 0 reactions 1 assignee Claimed by @llxia View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
33
Forks
97
Avg merge
7h 9m
Merged PRs (30d)
5

Description

With the current code, we are using a single node when running TRSS server. As we monitor more and more projects, the TRSS server can be very busy to process the Jenkins outputs. As a result, TRSS server may not be able to response the client requests in a timely manner.

With the current structure, we can easily break into two node servers:
- `frontend` - for responding to client requests (mainly for querying database)
- `backend` - for querying Jenkins and inserting/updating database.

We do not want to use `fork()` here because it will complicate the logic and add lots of if conditions. And these two tasks are very different.

Once two node servers are created, we can further improve the efficiency by leveraging multiple processors on the machine. We can use `fork()` to create multiple `backend` workers and we can process Jenkins jobs in parallel.

Note: a flag is needed to keep the worker id and to keep track of which job is processing by which `backend` worker. Also, we should set a timeout after which we assume the worker is dead and we can restart the worker.

![image](https://user-images.githubusercontent.com/19273206/57476879-89c28200-7265-11e9-9c0a-21256d54a3c8.png)

In summary, there are the following steps:

- [x] Break node server into two: `frontend` and `backend`

- [x] configure forever service on the server

- [ ] create multiple `backend` workers

- [ ] update readme

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.