Extending the file input plugin for checksum validation by calling a given REST endpoint
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
Hi,
I have extended the default **logstash-input-file** plugin. It takes as extra parameters an _authentication endpoint_ and _authentication parameters_. Then the plugin whenever a new file is identified calculates the MD5 checksum and calls the given authentication endpoint along with any other parameters given.
Based on the response code from the authentication endpoint the file is allowd to be processed or is disallowed.
```javascript
file_extended {
type => "some_type"
path => "/*.csv"
start_position => end
sincedb_path => "/SINCEDB.out"
auth_endpoint => "http://localhost:9090/microservice/validate"
auth_params => [ "customer=DINEMORE", "fileName=shabirmean", "canada", "mcgill"]
}
```
The default **file** plugin has been extended to include two additional configs:
* auth_endpoint
* auth_params
And all the other code changes have been done in the [ruby-filewatch](https://github.com/jordansissel/ruby-filewatch) repo which is used by the default plugin to monitor file changes.
@jordansissel: The **filewatch** library checks if the _auth_endpoint_ parameter is present and if so does the validation logic I had written and if not it behaves like the normal way it used to.
For the parameters in **auth_params**: if the param strings contain an *"="* (ie: in the form _name=value_) then they are appended as query strings and the rest are appended as path params. Hence, the validation endpoint needs to be written accordingly and also the **MD5 Checksum** is sent to the endpoint in the query string **"md5"**.
I wanted to see whether this will be of any use to the community and whether I can contribute it. Plus want some review on whether the way I had handled it is generic and accurate.
Thanks
Shabir
Contributor guide
Assessment
This issue has not been assessed yet.