out_file: By symlink_path, some symlinks are not created with bulk inputs
- Dominant language
- Ruby
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 20
Description
### Describe the bug
We are using the symlink_path setting with placeholders of the out_file plugin. However, symlinks are not created as expected for all buffer chunks (the latest chunk for each chunk key). In practice, when data containing multiple metadata is input at once (e.g., records with different myid values that chunk keys includes), only one symlink is created, and no others are created thereafter.
### To Reproduce
I used a docker image `fluentd:v1.19.0-1.0` in the experiment.
1. Run fluentd with configs below.
2. Input multiple data at once: `echo -e '{"myid":"aa","value":"111"}\n{"myid":"bb","value":"222"}' >> /tmp/test.log`
3. `data.bb.new.log` (the symlink for bb) is only created. /fluentd/log/buffer/ has two buffer files and two buffer meta files.
4. Input data with `myid=aa`: `echo -e '{"myid":"aa","value":"333"}' >> /tmp/test.log`
5. `data.aa.new.log` (the symlink for aa) is not created.
Configure file:
```
@type tail
path /tmp/test.log
tag test
@type json
@type file
@id output1
path /fluentd/log/data.${myid}.*.log
symlink_path /fluentd/log/data.${myid}.new.log
append true
@type file
timekey 1d
timekey_wait 1m
path /fluentd/log/buffer
```
### Expected behavior
A symlink is created for the latest chunk for each chunk key.
### Your Environment
```markdown
- Fluentd version: 1.19.0
- Package version: Docker image `fluentd:v1.19.0-1.0`
- Operating system:
- Kernel version:
(in actual operation: td-agent 4.3.2 fluentd 1.14.6 (c0f48a0080550eff6aa6fa19d269e480684e7a45))
```
### Your Configuration
```apache
Specified in “To Reproduce” section.
```
### Your Error Log
```shell
N/A
```
### Additional context
I tried to trace the relevant code.
`SymlinkBufferMixin#metadata` is implemented to hold a single `@latest_metadata`. It seems that this is held to use in the next `generate_chunk` call. I suspect that `metadata` might be called with different parameters before `generate_chunk` is called.
https://github.com/fluent/fluentd/blob/ed15edc4b7c86c1fcea3fbf326b7c42325e6fc71/lib/fluent/plugin/out_file.rb#L75-L84
Contributor guide
Assessment
This issue has not been assessed yet.