fluent / fluent/fluent-plugin-webhdfs
httpFS - Do not create file if it does not exist
- Dominant language
- Ruby
- Stars
- 60
- Forks
- 35
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
We are running a MapR custer and webHDFS is not supported by MapR. So we are trying to populate hadoop using httpFS.
Our Webhdfs config :
```
@type webhdfs
host mapr-mapr-master-0
port 14000
path "/uhalogs/docker/docker-%M.log"
time_slice_format %M
flush_interval 5s
username mapr
httpfs true
```
However when using the fluentd plugin, logs are appended correclty to an existing file. But if the file does not exist (using a timestamp-based filename), we get a WebHDFS::ServerError instead of a WebHDFS::FileNotFoundError that would create the file I guess.
Error 500 received by Mapr :
```
{
"RemoteException": {
"message": "Append failed for file: /uhalogs/docker/testfile.log, error: No such file or directory (2)",
"exception": "IOException",
"javaClassName": "java.io.IOException"
}
}
```
logs by fluentd-webhdfs plugin :
```
2017-01-12 13:59:09 +0000 [warn]: failed to communicate hdfs cluster, path: /uhalogs/docker/docker-58.log
2017-01-12 13:59:09 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-01-12 14:00:13 +0000 error_class="WebHDFS::ServerError" error="{\"RemoteException\":{\"message\":\"Append failed for file: \\/uhalogs\\/docker\\/docker-58.log, error: No such file or directory (2)\",\"exception\":\"IOException\",\"javaClassName\":\"java.io.IOException\"}}" plugin_id="object:3fe5f920c960"
2017-01-12 13:59:09 +0000 [warn]: suppressed same stacktrace
```
related code :
[https://github.com/fluent/fluent-plugin-webhdfs/blob/master/lib/fluent/plugin/out_webhdfs.rb#L262](url)
What I am not sure and I can't find proper specifications for HttpFS on the web is :
- Is it a bad implementation of httpFS on MapR side or should we handle this exception as well on the fluentd plugin ?
Thank You
Alban
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at lib/fluent/plugin/out_webhdfs.rb around line 262 and inspect how a missing file is handled when httpfs is enabled. Compare the MapR error response with the plugin's expected WebHDFS errors; the issue is done when the correct handling is established and covered for timestamp-based files that do not yet exist.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, ruby
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100