owncloud / owncloud/ocis

Postprocessing service might stop working due to a possible deadlock.

Open
#10,317 0 comments 0 reactions 0 assignees View on GitHub
Type:Bug
Dominant language
Go
Stars
2.1k
Forks
274
Avg merge
2d 1h
Merged PRs (30d)
103

Description

## Describe the bug
Right now, the postprocessing service is using only one thread to process the events. Processing these events might require the postprocessing service to retrieve/store data or to send other events. These actions are done against other services, which might take a long time to respond.

On the best case, this could be seen as a temporary hiccup. The postprocessing service would eventually get a network error (timeout or deadline reached) after several seconds without a response, so it should eventually process the next events.
On the worst case, that network error might not happen if there is no timeout associated to the connection, so it could wait indefinitely for a response. Even if we assume the processing is done by a different new thread, this could lead to a memory leak.

Regarding the events being processed, the `ResumePostprocessing` seems pretty bad because it needs to list what's in the store. Since we aren't putting any limitation (at least for now), we can assume that the listing will be very expensive. The problem is that the postprocessing service will be blocked until it finishes the operation.

## Steps to reproduce
No steps to reproduce because it's expected to happen on very heavy loads. Under normal load, and with a stable network and services, the postprocessing service should process the events quite fast.
1.
2.
3.

## Expected behavior
If there are no dependencies among any event, the postprocessing service can offload the processing of the events to a worker pool. The only option to reach the deadlock in this case is if all the workers are stuck waiting. This could happen with unstable services, but it should make the service more reliable in terms of network connection.

For the event processing, it's expected that the postprocessing services keeps reading events until all the workers are busy. This means that, even if one worker takes a lot of time to process an event, the rest of the workers can keep processing more events. This should work quite well for the `ResumePostprocessing` event, which is expected the block a worker for some time.

In addition, we need to make sure there are timeouts working for the connection. We need a way to ensure the workers eventually finish the job and can start with the next one.

## Actual behavior
The postprocessing service handles the processing all by itself.

## Setup
Please describe how you started the server and provide a list of relevant environment variables or configuration files.

```console
OCIS_XXX=somevalue
OCIS_YYY=somevalue
PROXY_XXX=somevalue
```

## Additional context
Based on findings for https://github.com/owncloud/ocis/issues/10220#issuecomment-2411328773 (this might NOT fix the linked issue)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.