Logstash crashing with exception IOError: Too many open files

We are running Logstash 2.1.3 on Ubuntu 14.04.4 and we are seeing crashes about once a week.

EDIT: Just starting the process makes it run for another week before crashing again.

This is from /var/log/logstash/logstash.err.1

```
IOError: Too many open files
select at org/jruby/RubyIO.java:3682
request at /opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:659
each_resource at /opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:512
resolv at /opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:1035
each at org/jruby/RubyArray.java:1613
resolv at /opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:1034
each at org/jruby/RubyArray.java:1613
resolv at /opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:1033
each at org/jruby/RubyArray.java:1613
resolv at /opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:1031
each_resource at /opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:503
each_resource at /opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:502
each_name at /opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:453
each_name at /opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:151
each at org/jruby/RubyArray.java:1613
each_name at /opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:150
getname at /opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:132
reverse at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dns-2.0.2/lib/logstash/filters/dns.rb:174
each at org/jruby/RubyArray.java:1613
reverse at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dns-2.0.2/lib/logstash/filters/dns.rb:155
filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dns-2.0.2/lib/logstash/filters/dns.rb:85
timeout at org/jruby/ext/timeout/Timeout.java:115
filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dns-2.0.2/lib/logstash/filters/dns.rb:84
multi_filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/filters/base.rb:151
each at org/jruby/RubyArray.java:1613
multi_filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/filters/base.rb:148
cond_func_91 at (eval):3073
each at org/jruby/RubyArray.java:1613
cond_func_91 at (eval):3069
filter_func at (eval):954
filterworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/pipeline.rb:244
start_filters at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/pipeline.rb:178
```

This is the limits for the running logstash process.

```
$ cat /proc/57469/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 15786 15786 processes
Max open files 16384 16384 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15786 15786 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
```

Start by tracing the DNS filter at logstash-filter-dns-2.0.2/lib/logstash/filters/dns.rb and the Ruby resolver calls shown in the stack trace, then inspect Logstash's pipeline handling in logstash-core-2.1.3-java/lib/logstash/pipeline.rb. Reproduce or monitor file-descriptor usage for the long-running process and determine what causes the limit to be exhausted; done means the crash cause and a verified fix or limitation are documented.

Written by the indexing model from the issue text.