aws / aws/containers-roadmap

[EKS] [request]: Implement workaround for EKS worker nodes hitting inotify watch limit interacting with Aurora Postgres

Open
#1,243 0 comments 0 reactions 0 assignees View on GitHub
EKS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
What do you want us to build?
Either include the identified workaround below in default EKS worker node AMIs, or find another way to avoid the error.

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We're running this query in one of our Aurora Postgres instances:

```
select
s.geoid10,
st_union(s.intersect_geom),
sum(s.intersect_area) as sidewalk_area,
(p.dp0010001 * 1.027) as population_2019
from
ny_sidewalk_intersects s
join ny_population_tracts_2010 p on (s.geoid10 = p.geoid10)
group by 1, 4
limit 1;
```

And getting this error: `Query 1 ERROR: ERROR: could not write block 7539461 of temporary file: No space left on device`

**Are you currently working around this issue?**
How are you currently solving this problem?
We've had to create a cloud-init script for our ASG launch configuration that sets the following sysctl parameter in order to work around it. Without this, we get EKS nodes hitting the kernel inotify watch limit every couple of weeks at least.

The workaround we put in cloud-init is:

sysctl fs.inotify.max_user_watches=524288

**Additional context**
Anything else we should know?

**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

Contributor guide

Open the contributing guide

Research direction

The issue concerns EKS worker node AMIs, an ASG cloud-init script, and the fs.inotify.max_user_watches sysctl setting. Start by reviewing how default EKS worker node AMIs configure kernel parameters and how the reported Aurora PostgreSQL workload triggers the limit. Done means the workaround is included by default or the error is avoided through another documented solution.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, postgresql, shell
Domain
cloud, databases, infrastructure, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.