geopython / geopython/pywps

Async processes accumulate until failure with sqlite in-memory

Open
#495 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
186
Forks
117
Avg merge
11d 20h
Merged PRs (30d)
1

Description

# Description
This relates to running processes in async mode, with `logging.database=sqlite:///:memory:`

When an async process completes, i.e. (pywps_requests.percent_done: 100.0, pywps_requests.status: 4) the `running_count` reported by [dblog](https://github.com/geopython/pywps/blob/034b185c5e3b521d4a6a30fef118c6ab368567a6/pywps/dblog.py#L88) never seems to reflects this.

So if I set `parallelprocesses=5` I can execute 5 successful jobs, however each job increments this running count, which is never decremented on completion, meaning I can only run 5 before all I get is ‘PyWPS Process GetSubset accepted’ response for a process which never runs.

This issue only seems to happen when using in-memory sqlite (i.e. does not occur when supplying my own sqlite db string).

# Environment

- operating system: Debian
- Python version: 3.6.5
- PyWPS version: 4.2.2
- source/distribution
- [x] git clone
- [ ] Debian
- [ ] PyPI
- [ ] zip/tar.gz
- [ ] other (please specify):
- web server
- [ ] Apache/mod_wsgi
- [ ] CGI
- [x] other (please specify): nginx/gunicorn

# Steps to Reproduce
- Run pywps configured with
- `logging.database=sqlite:///:memory:`
- `server.parallelprocesses=5`
- Submit 5 async requests
- i.e. with `storeExecuteResponse=true&status=true`
- should work as expected, then on 6th request:
- [Gets put in the stored queue](https://github.com/geopython/pywps/blob/034b185c5e3b521d4a6a30fef118c6ab368567a6/pywps/app/Process.py#L167)
- despite status being set to [PyWPS Process stored in job queue](https://github.com/geopython/pywps/blob/034b185c5e3b521d4a6a30fef118c6ab368567a6/pywps/app/Process.py#L173), the response document still says ‘PyWPS Process GetSubset accepted’
- nothing else can run now

# Additional Information
This is being run in a docker container from macos host, don’t _think_ it should affect this 🤔

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.