buf_file: queued file chunks keep FDs open until shutdown; resume opens all chunks — EMFILE death spiral on slow output

### Describe the bug

When using `@type file` buffers with a slow or failing output (S3, Kafka, forward, etc.), Fluentd accumulates many buffer chunk files. Each chunk keeps one or two file descriptors open while it sits in **stage** or **queue**, and `buf_file#resume` re-opens every persisted chunk on startup. Once open FD count approaches the process limit (often 65536 on Linux), Fluentd enters a **death spiral**:

1. Output flush fails with `Errno::EMFILE` (cannot open TCP socket / DNS / temp files)
2. Queued chunks are not purged and their FDs are not released
3. New chunks cannot be created (`BufferOverflowError`)
4. Even after downstream recovers, Fluentd may remain stuck because resume/flush paths need more FDs than available

This is related to long-standing reports ([#1612](https://github.com/fluent/fluentd/issues/1612), [#3040](https://github.com/fluent/fluentd/issues/3040), [#4012](https://github.com/fluent/fluentd/issues/4012), [#3993](https://github.com/fluent/fluentd/issues/3993)), but those issues focus on symptoms/workarounds (raise ulimit, tune buffer size). This report points at a specific **FD lifecycle design** in current `buf_file` / `FileChunk` that makes recovery impossible at scale without deleting buffers or raising limits to match chunk count.

### Root cause in code (v1.x)

**1. `enqueue_chunk` closes only empty chunks — non-empty queued chunks stay open**

```ruby
# lib/fluent/plugin/buffer.rb
if chunk.empty?
chunk.close
else
@queue << chunk
chunk.enqueued! # renames files but keeps @chunk and @meta open
end
```

**2. `FileChunk#enqueued!` keeps both `@chunk` and `@meta` handles open after rename**

After staging, each chunk uses **2 FDs** (`@chunk` + `@meta`). After enqueue, both handles remain open through flush (see `file_rename` callbacks in `lib/fluent/plugin/buffer/file_chunk.rb`). The code even notes:

> "Too many open files" should be fixed by proper buffer configuration and system setting.

**3. `buf_file#resume` opens every buffer file during startup**

```ruby
# lib/fluent/plugin/buf_file.rb
Dir.glob(escaped_patterns(patterns)) do |path|
chunk = Fluent::Plugin::Buffer::FileChunk.new(m, path, mode, ...)
queue << chunk # File.open in load_existing_enqueued_chunk / load_existing_staged_chunk
end
```

With tens of thousands of backlog files, resume alone can exhaust FDs **before any flush thread runs** ([#3040](https://github.com/fluent/fluentd/issues/3040)).

**4. FDs are only released on buffer shutdown**

`Buffer#close` closes dequeued, queued, and staged chunks — not when a chunk moves from stage → queue.

PR [#1468](https://github.com/fluent/fluentd/pull/1468) (v0.14.12) handles EMFILE on **chunk create** by raising `BufferOverflowError`, but does not close existing queued chunk FDs or enable recovery.

### To Reproduce

1. Configure a file buffer with relatively small `timekey` and large `chunk_limit_size`, e.g. S3 output:

```apache

@type s3
# ... aws / bucket config ...

@type file
path /var/log/fluent/buffer
timekey 5
chunk_limit_size 100MB
flush_mode interval
flush_interval 5
flush_thread_count 15
# note: no total_limit_size — backlog can grow by chunk count

```

2. Block or fail the output path (wrong credentials, network partition, unreachable S3 endpoint) for long enough to accumulate **~30k+ chunks** (≈ 60k+ FDs with chunk+meta open, or ~32k chunks at 65536 limit).

3. Observe logs:

```
failed to flush the buffer. retry_times=... error="Too many open files" ...
Failed to open TCP connection to ...s3.amazonaws.com:443
can't create buffer metadata for ... error = Too many open files @ rb_sysopen
```

4. Restore downstream — Fluentd often **does not recover** without manual intervention (delete buffer dir or raise `LimitNOFILE` above chunk count).

5. Restart Fluentd with large backlog — `restoring buffer file` storm in logs; process hits EMFILE during `resume` before flushing ([#3040](https://github.com/fluent/fluentd/issues/3040)).

### Expected behavior

File buffer chunks should not require a permanently open FD for every queued chunk. Suggested approaches (any of these would help):

1. **Close on enqueue**: after `enqueued!`, call `chunk.close` (or equivalent) so queued chunks hold **0 FDs** until dequeued for flush.
2. **Lazy open on flush**: `FileChunk#open` / `read` reopen paths on demand; keep metadata in memory or mmap only `.meta` briefly.
3. **Lazy resume**: do not `File.open` all chunks in `Dir.glob`; index paths from filesystem and open in batches bounded by `(flush_thread_count * 2)` or configurable `resume_open_limit`.
4. **Document FD estimation**: `queue_limit_length`, `total_limit_size`, and chunk count vs `LimitNOFILE` (as suggested in [#3040](https://github.com/fluent/fluentd/issues/3040#issuecomment)).

At minimum, Fluentd should remain able to **drain an existing backlog** after downstream recovery without requiring FD count ≥ chunk count.

### Actual behavior

- ~1–2 FDs per staged/queued chunk for the lifetime of the chunk in memory
- Resume opens all chunks eagerly
- EMFILE on output flush → permanent stall until buffer deletion or ulimit >> chunk count
- `BufferOverflowError` on create stops ingestion but does not free queued FDs

### Your Environment

```markdown
- Fluentd: 1.16+ / td-agent 4.x (behavior present since v0.14 file buffer redesign)
- OS: Linux (Kubernetes), `LimitNOFILE` commonly 65536
- Buffer: `@type file` with timekey slicing, S3 or other slow output
- Observed at production scale: ~65k FD limit reached with ~32k queued chunks (chunk + meta open)
```

### Your Error Log

```
failed to flush the buffer. retry_times=0 next_retry_time=... error="Too many open files"
error_class=Errno::EMFILE
Failed to open TCP connection to .s3.amazonaws.com:443
```

```
emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError
error="can't create buffer metadata for /var/log/fluent/buffer/buffer.*.log.
Stop creating buffer files: error = Too many open files @ rb_sysopen - ...log.meta"
```

### Additional context

- Community has reported this pattern since 2017 ([#1612](https://github.com/fluent/fluentd/issues/1612): 131034 files, 65516 `.log` + 65515 `.meta`, exactly at 65536 ulimit).
- [#3040](https://github.com/fluent/fluentd/issues/3040) (still open) requests batch open/close on resume — no upstream fix yet.
- Workarounds today: raise ulimit, delete buffer files (data loss), or cap backlog via `total_limit_size` / `overflow_action` — none fix FD retention on queued chunks.
- Willing to contribute a PR for close-on-enqueue + lazy reopen if maintainers agree on direction.

Start by reading lib/fluent/plugin/buffer.rb, lib/fluent/plugin/buffer/file_chunk.rb, and lib/fluent/plugin/buf_file.rb, focusing on enqueue, resume, and close behavior. Reproduce the EMFILE scenario with a large queued file-buffer backlog; done requires an agreed lifecycle change that lets Fluentd recover and drain the backlog without requiring one or two open FDs per chunk.

Written by the indexing model from the issue text.