hasura / hasura/graphql-engine

Disk IOPS growing out of control as event tables grow

Open
#7,028 6 comments 0 reactions 0 assignees View on GitHub
a/eventing/data-triggers
Dominant language
TypeScript
Stars
32.1k
Forks
3k
PR merge metrics
PR metrics pending

Description

We run our database on RDS (AWS) using their SSD storage option. As described [here](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html), I/O operations are a budgeted resource with a slowly-replenishing credit balance. If you run out of IOPS credits, your performance tanks.

Recently our Postgres read IOPS has started growing out of control. I wasn't sure how to investigate this (tips are welcome!) but we eventually truncated `hdb_catalog.event_invocation_logs` and dropped all delivered || errored rows from `hdb_catalog.event_log`. read IOPS dropped drastically.

I think this may be a documentation bug; there is no expectation that Hasura Actions perform well when a core table reaches >1M rows, but I think it would be nice to discuss the specific risk somewhere in the documentation.

I'm also wondering what specific mechanism is causing this huge amount of I/O reads. In RDS Performance Insights we saw that the following query was spending a lot of time waiting on `DataFileRead`, which suggests it may have been causing a lot of read I/O:

```sql
UPDATE hdb_catalog.event_log
SET locked = ?
WHERE id IN ( SELECT l.id
FROM hdb_catalog.event_log l
WHERE l.delivered = ? and l.error = ? and l.locked = ?
and (l.next_retry_at is NULL or l.next_retry_at <= now())
and l.archived = ?
ORDER BY created_at
LIMIT $1
FOR UPDATE SKIP LOCKED )
RETURNING id, schema_name, table_name,
```

I'm thinking maybe the problem is lock contention?

Contributor guide

Open the contributing guide

Research direction

Start with the reported UPDATE query on hdb_catalog.event_log and the linked AWS RDS storage documentation. Investigate the reported DataFileRead activity alongside growth in hdb_catalog.event_invocation_logs and event_log. Done means identifying or narrowing the I/O risk and documenting the relevant table-growth limits and operational guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, postgres
Domain
cloud, databases, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.