Some DataFusion compactions hang
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
Had an ingest running over 5 days. Observed 11 of the ~911,000 compactions in that time requiring two runs to complete. The compaction tracker would show the first runs as started, but no finish or error message. The second runs would always be started roughly 12 hours later and would finish successfully with an average duration of roughly 3 minutes. So no messages on the compactions DLQ.
Despite no outstanding compactions, found 11 Fargate compaction tasks still running. Their start times correlated with the start times of the 11 compactions. Logs showed that they had each picked up one of those compactions and had started processing it using DataFusion - logs lines included ‘row and sort key field order’ and ‘applying sleeper aggregation’. However, there weren't any DataFusion logs lines from ‘sketch.rs’ or ‘metrics.rs’ which are normally seen afterwards. The only logging seen after this was periodic 'ChangeMessageVisibilityTimeoutAction' messages every 5 mins. These continued for roughly 12 hours until an 'SqsException: Value 900 for parameter VisibilityTimeout is invalid. Reason: Total VisibilityTimeout for the message is beyond the limit [43200 seconds]' error message. There are no logs after this, but the containers continued to run. After 12 hours, SQS would have made the compaction job messages visible again on the queue. They will have then been picked up by a new Fargate task and processed successfully the second time.
Hard to reproduce this issue reliably as it only affected a small proportion of compactions. Some of the compaction tasks (8) had processed other compactions before hanging, some of them (3) were hanging on the first compaction that they attempted.
### Expected behaviour
DataFusion should not hang when processing compactions.
Fargate tasks should not run indefinitely.
Failing to extend the compaction message visibility should maybe cause the container to force quit as the job will get picked up by another container.
Contributor guide
Assessment
This issue has not been assessed yet.