lablup / lablup/backend.ai

Add progress unit hints to background task framework

Open
#381 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

(Note: it is desirable to complete #306 first before doing this issue.)

- [ ] Let's add optional kwargs to `BackgroundTask.start()` so that bgtask creators can specify the availability of progress information and its unit. If not given, treat as unavailable.
- [ ] Let's add the following extra information as extension HTTP headers in the SSE reply:\* Include whether the bgtask provides progress information: `X-BackendAI-Progress-Available`. The values may be `yes` or `no`.
- Include the progress unit hint: `X-BackendAI-Progress-Unit`. The values may be `bytes`, `files`, `items`, `ops` and there may be task-specific ones (e.g., `layers` for kernel pull progress).
- NOTE: The reason to use HTTP headers is to allow reading them before setting up the SSE (server-sent events) message fetching loop.

- [ ] Refactor the implementation of the background task framework\* Use `aiotools.PersistentTaskGroup` to keep track of asyncio tasks and gracefully shutdown.
- Update the usage of Redis pipelines, referring [the common's new test cases](https://github.com/lablup/backend.ai-common/blob/main/tests/redis/test_pipeline.py).

- [ ] In the client & client-cli, we need to utilize the progress hint as follows:\* FYI: The extension headers will be available in the response object's `headers` field.
- Pass `disable=True` to the `tqdm` constructor when progress information is unavailable.
- Fallback to the existing implementation if the headers are not availble for backward compatibility.
- The update is required for `admin image rescan` and `vfolder clone`. (Please add if I've missed others!)
- Also apply to #227.

JIRA Issue: BA-275

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.