gchq / gchq/sleeper

Some DataFusion compactions hang

Open
#7,053 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

Had an ingest running over 5 days. Observed 11 of the ~911,000 compactions in that time requiring two runs to complete. The compaction tracker would show the first runs as started, but no finish or error message. The second runs would always be started roughly 12 hours later and would finish successfully with an average duration of roughly 3 minutes. So no messages on the compactions DLQ.

Despite no outstanding compactions, found 11 Fargate compaction tasks still running. Their start times correlated with the start times of the 11 compactions. Logs showed that they had each picked up one of those compactions and had started processing it using DataFusion - logs lines included ‘row and sort key field order’ and ‘applying sleeper aggregation’. However, there weren't any DataFusion logs lines from ‘sketch.rs’ or ‘metrics.rs’ which are normally seen afterwards. The only logging seen after this was periodic 'ChangeMessageVisibilityTimeoutAction' messages every 5 mins. These continued for roughly 12 hours until an 'SqsException: Value 900 for parameter VisibilityTimeout is invalid. Reason: Total VisibilityTimeout for the message is beyond the limit [43200 seconds]' error message. There are no logs after this, but the containers continued to run. After 12 hours, SQS would have made the compaction job messages visible again on the queue. They will have then been picked up by a new Fargate task and processed successfully the second time.

Hard to reproduce this issue reliably as it only affected a small proportion of compactions. Some of the compaction tasks (8) had processed other compactions before hanging, some of them (3) were hanging on the first compaction that they attempted.

### Expected behaviour

DataFusion should not hang when processing compactions.
Fargate tasks should not run indefinitely.
Failing to extend the compaction message visibility should maybe cause the container to force quit as the job will get picked up by another container.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.