[EKS] [request]: Implement workaround for EKS worker nodes hitting inotify watch limit interacting with Aurora Postgres
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
What do you want us to build?
Either include the identified workaround below in default EKS worker node AMIs, or find another way to avoid the error.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We're running this query in one of our Aurora Postgres instances:
```
select
s.geoid10,
st_union(s.intersect_geom),
sum(s.intersect_area) as sidewalk_area,
(p.dp0010001 * 1.027) as population_2019
from
ny_sidewalk_intersects s
join ny_population_tracts_2010 p on (s.geoid10 = p.geoid10)
group by 1, 4
limit 1;
```
And getting this error: `Query 1 ERROR: ERROR: could not write block 7539461 of temporary file: No space left on device`
**Are you currently working around this issue?**
How are you currently solving this problem?
We've had to create a cloud-init script for our ASG launch configuration that sets the following sysctl parameter in order to work around it. Without this, we get EKS nodes hitting the kernel inotify watch limit every couple of weeks at least.
The workaround we put in cloud-init is:
sysctl fs.inotify.max_user_watches=524288
**Additional context**
Anything else we should know?
**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)
Contributor guide
Research direction
The issue concerns EKS worker node AMIs, an ASG cloud-init script, and the fs.inotify.max_user_watches sysctl setting. Start by reviewing how default EKS worker node AMIs configure kernel parameters and how the reported Aurora PostgreSQL workload triggers the limit. Done means the workaround is included by default or the error is avoided through another documented solution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, postgresql, shell
- Domain
- cloud, databases, infrastructure, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100