apache / apache/jena

Fuseki 6.0.0: canceled federated SERVICE queries can wedge the target dataset until restart

Open
#3,837 10 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.4k
Forks
712
Avg merge
15h 41m
Merged PRs (30d)
53

Description

### Version

6.0.0 Also reproduced on 5.5.0.

### What happened?

### Version

6.0.0

Also reproduced on 5.5.0.

### What happened?

We can reproduce a failure mode where repeated canceled federated `SERVICE` queries leave
the target dataset effectively wedged until Fuseki is restarted.

The pattern is:

1. A direct query to dataset `target` succeeds.
2. A federated query from dataset `source` to dataset `target` using `SERVICE
` succeeds.
3. We then issue a burst of heavy federated queries from `source` to `target`, with the
client canceling/timing out the outer HTTP request almost immediately.
4. After that, even a simple direct query to dataset `target` times out.
5. Restarting Fuseki clears the problem.

This is reproducible for us on Jena/Fuseki 6.0.0 and also on 5.5.0.

### Why this looks distinct from ordinary timeout behavior

This does not look like only the outer client timing out.

After the cancellation storm:
- a direct query to the target dataset also times out
- the store recovers only after restart

So the target dataset/server appears to be left in a bad runtime state.

### Reproducing it

We reproduced this against an isolated standalone Fuseki 6.0.0 container built from the
official Apache release tarball.

Our production datasets are private, but the failure can be described with this structure:

- source dataset: `source`
- target dataset: `target`

Baseline direct probe against `target`:

```sparql
SELECT * WHERE {
?p ?o
}
LIMIT 5
```
Baseline federated probe from source to target:
```sparql
SELECT * WHERE {
SERVICE {
?p ?o
}
}
LIMIT 5
```
Cancellation-storm query:
```sparql
SELECT * WHERE {
SERVICE {
?s ?p ?o
}
}
```
We then repeatedly send that last query to the source dataset and cancel the outer HTTP
request almost immediately, for example:

```sh
for i in $(seq 1 40); do
curl -sS --max-time 0.05 -G \
--data-urlencode 'query=SELECT * WHERE { SERVICE { ?s ?p ?o } }' \
http://127.0.0.1:3030/source/sparql >/dev/null || true
done
```

### Actual result

Before stress:

- direct query succeeds
- federated query succeeds

After the canceled federated-query burst:

- direct query to target times out
- federated query also fails/times out
- Fuseki restart is required to recover

### Expected result

Canceled outer federated queries should not leave the target dataset/server wedged.
After the canceled requests, normal direct queries to the target dataset should still work.

### Relevant logs

From the Jena 6.0.0 Fuseki log, after the stress starts we see many inner requests like:
```
GET http://127.0.0.1:3030/target/sparql?query=SELECT++%2A%0AWHERE%0A++%7B+?s++?p++?o+%7D%0A
```
The outer requests are being canceled by the client, but the inner SERVICE subqueries
continue to run. After enough of these, the target dataset stops responding to even direct
queries.

### Notes

- This was reproduced using loopback 127.0.0.1, so it does not appear to require Docker DNS/
container-name routing.
- We specifically tested 6.0.0 because the changelog mentions query-cancellation
improvements, but we still reproduce this failure.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the cancellation storm against standalone Fuseki 6.0.0 with the supplied curl loop, then inspect the logs showing inner SERVICE requests continuing after outer cancellation. The fix is done when repeated canceled federated queries leave the target responsive to direct queries without a restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.