Document various platform-imposed file-descriptor limits that might be encountered in a Twisted-using application

	tomsheffler reported
Trac ID	trac#5337
Type	enhancement
Created	2011-10-26 13:04:29Z

My application encountered "Could not accept new connection (EMFILE)" tracebacks. (EMFILE is too many open file descriptors).

I used lsof to see open connections and could see that there were a few clients that opened many connections without closing their previous ones. I think I need to protect against this in my own protocol handling, so I don't consider this an error in twisted.

I counted the total number of open files from lsof: it was approx 1080. With the epoll reactor I was assuming that this limit would not be hit, because I assumed choosing the epoll reactor would have unlimited connections.

I now believe that I need to raise the RLIMIT_NOFILE myself if I want the limit to be higher. Perhaps in my .tac file. Probably something like this:

resource.setrlimit(resource.RLIMIT_NOFILE, (10000, 10000))

FYI - On my LInux system, "ulimit -n" reports 1024.

I haven't found any documentation that says when/how you should raise the RLIMIT_NOFILE and how it interacts with epoll. It would be nice if this were mentioned somewhere. Some tests refer RLIMIT_NOFILE, but the use is unclear.

trac-id__5337 5337
type__enhancement enhancement
reporter__tomsheffler tomsheffler
priority__normal normal
milestone__ 
branch__ 
branch_author__ 
status__new new
resolution__None None
component__core core
keywords__documentation documentation
time__1319634269000000 1319634269000000
changetime__1319634659000000 1319634659000000
version__None None
owner__ 

The issue names no documentation file; start by reviewing the existing tests that refer to RLIMIT_NOFILE and the epoll reactor, using the Linux ulimit -n and .tac examples as context. Document how platform file-descriptor limits relate to epoll and when an application may need to raise RLIMIT_NOFILE; done when this guidance is available in the appropriate Twisted documentation.