elastic / elastic/logstash

Double, double quoting

Open
#3,878 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

While attempting to parse a perfectly legit CSV, we're getting exceptions like the following:

```
{:timestamp=>"2015-09-04T17:50:39.634000-0500", :message=>"Trouble parsing csv", :source=>"message", :raw=>"

 

", :exception=>#, :level=>:warn}
```

As you can see, our csv field has embedded html which requires the use of the double quote in its content... necessitating two double quotes in order to differentiate between the field start/end and content inside the field. The RFC explains it better. Per [RFC4180](https://tools.ietf.org/html/rfc4180#section-2) (paragraph 7),

> If double-quotes are used to enclose fields, then a double-quote
> appearing inside a field must be escaped by preceding it with
> another double quote.

The logstash csv filter doesn't seem to account for the double double-quote scenario and therefore ends a field prematurely and spits out these malformed csv errors. I honestly haven't had a time to dig into the logstash code but I'm thinking this is a bug.

This is forcing us to use the elasticsearch-csv-river when we'd rather move to logstash. Has this been identified before? Any suggested workarounds?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.