hasura / hasura/graphql-engine
Disk IOPS growing out of control as event tables grow
- Dominant language
- TypeScript
- Stars
- 32.1k
- Forks
- 3k
- PR merge metrics
- PR metrics pending
Description
We run our database on RDS (AWS) using their SSD storage option. As described [here](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html), I/O operations are a budgeted resource with a slowly-replenishing credit balance. If you run out of IOPS credits, your performance tanks.
Recently our Postgres read IOPS has started growing out of control. I wasn't sure how to investigate this (tips are welcome!) but we eventually truncated `hdb_catalog.event_invocation_logs` and dropped all delivered || errored rows from `hdb_catalog.event_log`. read IOPS dropped drastically.
I think this may be a documentation bug; there is no expectation that Hasura Actions perform well when a core table reaches >1M rows, but I think it would be nice to discuss the specific risk somewhere in the documentation.
I'm also wondering what specific mechanism is causing this huge amount of I/O reads. In RDS Performance Insights we saw that the following query was spending a lot of time waiting on `DataFileRead`, which suggests it may have been causing a lot of read I/O:
```sql
UPDATE hdb_catalog.event_log
SET locked = ?
WHERE id IN ( SELECT l.id
FROM hdb_catalog.event_log l
WHERE l.delivered = ? and l.error = ? and l.locked = ?
and (l.next_retry_at is NULL or l.next_retry_at <= now())
and l.archived = ?
ORDER BY created_at
LIMIT $1
FOR UPDATE SKIP LOCKED )
RETURNING id, schema_name, table_name,
```
I'm thinking maybe the problem is lock contention?
Contributor guide
Research direction
Start with the reported UPDATE query on hdb_catalog.event_log and the linked AWS RDS storage documentation. Investigate the reported DataFileRead activity alongside growth in hdb_catalog.event_invocation_logs and event_log. Done means identifying or narrowing the I/O risk and documenting the relevant table-growth limits and operational guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, postgres
- Domain
- cloud, databases, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100