elastic / elastic/logstash

input-csvfile for Consideration

Open
#3,777 2 comments 0 reactions 0 assignees View on GitHub
new plugin
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

I needed to process CSV files with first line schemas (that can vary from file-to-file) but see there's no clean way to do so with today's plugins. In some forum comments I see that improved CSV handling is on the "to do" list. I went ahead and coded-up a subclass of logstash-input-file that adds CSV parsing with a first-line-schema mode. The fundamental CSV parsing logic is largely borrowed from logstash-filter-csv, albeit extended with a first-line-schema mode.

I'd be interested in having it considered for submission as a standard plugin. Code/tests are at https://github.com/jweite/logstash-input-csvfile1, and particularly to get some other eyes on my code, as it's my first Logstash plugin.

PS, while I initially considered enhancing logstash-filter-csv I ultimately concluded that the only 100% reliable way to restart stream processing mid-file was to re-read the file's schema row, something that only the file input plugin can always do.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.