database size control and gc strategy proposal

1. Motivation

OpenPAI contains a postgresql db for job submitting, querying and other storage. The database leverages a limited size storage. If we don't do any gc, the size of storage will grow infinitely until the user cannot submit any new job.

2. Proposal

Disk usage control for non-critical function

We can classify all openpai function that relies on db into two categories:

• critical function: job submitting, job querying, job listing. Critical function uses the framework table.
• non-critical function: job retry history, job event, task retry history, ..., etc. Most non-critical functions use a different table, and have a foreign key which references framework.name.

For non-critical function, they should be sacrificed first before the critical function. Here we don't want use a delete-on-write policy or database trigger because it may cause issues like dead lock and low efficiency. Also, deleting user's record is always not a good practice. Instead, we set a max disk usage for these non-critical functions. If the disk usage reaches certain percentage, we disable these functions so the critical function won't be affected by them. Also, a disk usage alert should be sent to admin, which contains the table size and other useful information. Admin should address the issue ASAP.

e.g. We could set up a max disk usage percentage 80% for event collection. If the disk usage is > 80%, the event collection should stop. Users can submit new jobs because there is still 20% disk room for the critical function. But they cannot see events of the newly-submitted job. Admin will receive an alert about the situation.

We should prefer the non-critical service to implements this logic by themself. If it is not feasible, we can kill them from outside when the disk usage is reached.

Different non-critical functions may set different percentages. But we can use one percentage for all non-critical services in the beginning.

GC

For GC, we notice that most openapi resources (e.g. framework retry, pod, .. etc) in database reference the column framework.name. Thus we can delete some frameworks from framework table periodically to release some disk space. The most simple strategy will be delete frameworks older than N days. This GC strategy should be set by admin.

If the database usage doesn't reference framework.name, it is the service's own duty to set up a proper GC strategy.

Start by examining src/internal-storage and the database users of the framework table and framework.name references. Compare the proposed non-critical disk thresholds, admin alert contents, and age-based framework cleanup, then define completion as an agreed design for protecting critical operations and reclaiming storage.