Double, double quoting
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
While attempting to parse a perfectly legit CSV, we're getting exceptions like the following:
```
{:timestamp=>"2015-09-04T17:50:39.634000-0500", :message=>"Trouble parsing csv", :source=>"message", :raw=>"
", :exception=>#, :level=>:warn}
```
As you can see, our csv field has embedded html which requires the use of the double quote in its content... necessitating two double quotes in order to differentiate between the field start/end and content inside the field. The RFC explains it better. Per [RFC4180](https://tools.ietf.org/html/rfc4180#section-2) (paragraph 7),
> If double-quotes are used to enclose fields, then a double-quote
> appearing inside a field must be escaped by preceding it with
> another double quote.
The logstash csv filter doesn't seem to account for the double double-quote scenario and therefore ends a field prematurely and spits out these malformed csv errors. I honestly haven't had a time to dig into the logstash code but I'm thinking this is a bug.
This is forcing us to use the elasticsearch-csv-river when we'd rather move to logstash. Has this been identified before? Any suggested workarounds?
Contributor guide
Assessment
This issue has not been assessed yet.