Add progress unit hints to background task framework
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
(Note: it is desirable to complete #306 first before doing this issue.)
- [ ] Let's add optional kwargs to `BackgroundTask.start()` so that bgtask creators can specify the availability of progress information and its unit. If not given, treat as unavailable.
- [ ] Let's add the following extra information as extension HTTP headers in the SSE reply:\* Include whether the bgtask provides progress information: `X-BackendAI-Progress-Available`. The values may be `yes` or `no`.
- Include the progress unit hint: `X-BackendAI-Progress-Unit`. The values may be `bytes`, `files`, `items`, `ops` and there may be task-specific ones (e.g., `layers` for kernel pull progress).
- NOTE: The reason to use HTTP headers is to allow reading them before setting up the SSE (server-sent events) message fetching loop.
- [ ] Refactor the implementation of the background task framework\* Use `aiotools.PersistentTaskGroup` to keep track of asyncio tasks and gracefully shutdown.
- Update the usage of Redis pipelines, referring [the common's new test cases](https://github.com/lablup/backend.ai-common/blob/main/tests/redis/test_pipeline.py).
- [ ] In the client & client-cli, we need to utilize the progress hint as follows:\* FYI: The extension headers will be available in the response object's `headers` field.
- Pass `disable=True` to the `tqdm` constructor when progress information is unavailable.
- Fallback to the existing implementation if the headers are not availble for backward compatibility.
- The update is required for `admin image rescan` and `vfolder clone`. (Please add if I've missed others!)
- Also apply to #227.
JIRA Issue: BA-275
Contributor guide
Assessment
This issue has not been assessed yet.