fluent / fluent/fluentd

Output csv with header(columns)

Open
#3,070 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

**Is your feature request related to a problem? Please describe.**

I want fluentd to write a csv file with header like below.
(The header is written in only head of a file, of course.)

e.g.
```sample.csv
time,host,req_id,user
2013/02/28 12:00:00,192.168.0.1,111,-
2013/02/28 13:00:00,192.168.0.1,111,-
```

In my project, I must make a service outputs csv logs with headers like below example.
However, fluentd emit a record one by one that not to consider if it is header.
Is there future considering about handling this?

**Describe the solution you'd like**

No specific cool solution now. And I don't know this usage suites fluentd concepts.
Anyway, I have just written an ad hoc plugin extending [out_file](https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/out_file.rb) plugin. I'll explain it next item.

Well, I move on talk about practical implementation.
Would be nice to have some kind of "isHeader" if user need header.
I think it is OK with modifying out_file or making a new plugin handling it.
I hope we discuss that.

**Describe alternatives you've considered**

Here is the code snippet. I understand it is not cool.

```ruby
def write_without_compression(path, chunk)
File.open(path, "ab", @file_perm) do |f|
# Add below code block.
if f.stat.size == 0
# I cannot fetch "fields" from fluentd, so I just put it directly.
# This is because only formatter plugins contain "fields" information as object, I think.
f.write("time,host,req_id,user")
f.write("\n")
end
chunk.write_to(f)
end
end
```

This barely completes my demand.

**Additional context**
N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.