queue.process_queue is neither FIFO or LIFO
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
The queue.process_queue runner module returns items from the queue in what appears to be a lexical order as opposed to using either a FIFO or LIFO strategy.
Setup
setup a runner queue by adding the following to your master config
runner_queue:
queue: runners
backend: sqlite
Add any number of items to a queue:
salt-run queue.insert myqueue "a"
salt-run queue.insert myqueue "d"
salt-run queue.insert myqueue "z"
salt-run queue.insert myqueue "b"
Steps to Reproduce Issue
Use the queue.process_queue function to pop a number of items off the queue (but not the entire queue) or simply list the queue.
For example:
salt-run queue.process_queue myqueue quantity=2
In that example, items "a" and "b" would be popped off the queue and sent to the event bus.
Initially, this doesn't seem like it would be much of an issue, unless you add "a" and "b" back into the queue before "d" and "z" are popped off. If they are added back to the queue, then "d" and "z" will never be sent to the event bus as the 2 items always processed off the queue would be "a" and "b".
Both existing queue types (sqlite and pgjsonb) have an id field in queue tables created which could be used in selects to sort the data by how it was entered into the table ("ORDER BY id ASC" (for FIFO) or "ORDER BY id DESC" (for LIFO)). This should return the items in a more defined manner.
Versions Report
Tested/used on 2018.3, but code appears the same in newer versions
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting the queue.process_queue entry point and the sqlite and pgjsonb queue implementations mentioned in the report. Reproduce the issue with queue.insert and queue.process_queue quantity=2, then verify that partial processing follows a defined insertion order and that requeued items do not starve later entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100