Encrypt or hide ES credentials in td-agent configuration file
- Dominant language
- Ruby
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 20
Description
### Describe the bug
We want to encrypt the elasticsearch password mentioned in the td-agent configuration.
### To Reproduce
```xml
@type elasticsearch_dynamic
host
port 9200
logstash_format true
index_name ${'indexname'+'.'+Date.parse(record['Time']).strftime('%W%Y')}
time_key Time
logstash_format false
flush_interval 10s
user elastic
password
scheme https
client_cert /path-to-certificate
client_key /path-to-cert-key
ssl_version TLSv1_2
ssl_verify false
```
### Expected behavior
Encrypted elasticsearch password in the conf block
### Your Environment
```markdown
- Fluentd version: 1.14.3
- TD Agent version: td-agent 4.3.0
- Operating system: Ubuntu 18.04
- Kernel version: 4.15.0-192-generic
```
### Your Configuration
```apache
####
## Output descriptions:
##
# Treasure Data (http://www.treasure-data.com/) provides cloud based data
# analytics platform, which easily stores and processes data from td-agent.
# FREE plan is also provided.
# @see http://docs.fluentd.org/articles/http-to-td
#
# This section matches events whose tag is td.DATABASE.TABLE
@type tdlog
@id output_td
apikey YOUR_API_KEY
auto_create_table
@type file
path /var/log/td-agent/buffer/td
@type file
path /var/log/td-agent/failed_records
## match tag=debug.** and dump to console
@type stdout
@id output_stdout
####
## Source descriptions:
##
## built-in TCP input
## @see http://docs.fluentd.org/articles/in_forward
@type forward
@id input_forward
## built-in UNIX socket input
#
# type unix
#
# HTTP input
# POST http://localhost:8888/?json=
# POST http://localhost:8888/td.myapp.login?json={"user"%3A"me"}
# @see http://docs.fluentd.org/articles/in_http
@type http
@id input_http
port 8888
## live debugging agent
@type debug_agent
@id input_debug_agent
bind 127.0.0.1
port 24230
####
## Examples:
##
## File input
## read apache logs continuously and tags td.apache.access
#
# @type tail
#
# @type tail
# @id input_tail
#
# @type apache2
#
# path /var/log/httpd-access.log
# tag td.apache.access
#
## File output
## match tag=local.** and write to file
#
# @type file
# @id output_file
# path /var/log/td-agent/access
#
## Forwarding
## match tag=system.** and forward to another td-agent server
#
# @type forward
# @id output_system_forward
#
#
# host 192.168.0.11
#
# # secondary host is optional
#
#
# host 192.168.0.12
#
#
#
## Multiple output
## match tag=td.*.* and output to Treasure Data AND file
#
# @type copy
#
# @type copy
# @id output_copy
#
# @type tdlog
# apikey API_KEY
# auto_create_table
#
# @type file
# path /var/log/td-agent/buffer/td
#
#
#
# @type file
# path /var/log/td-agent/td-%Y-%m-%d/%H.log
#
#
@type elasticsearch_dynamic
host hostname
port 9200
include_tag_key true
tag_key @log_name
logstash_format true
index_name ${'index name'+'.'+Date.parse(record['Time']).strftime('%m%Y')}
time_key Time
logstash_format false
flush_interval 10s
user username
password password
scheme https
client_cert /path-to-certificate
client_key /path-to-certificate-key
ssl_version TLSv1_2
ssl_verify false
```
### Your Error Log
```shell
NA
```
### Additional context
We want to encrypt the password mentioned in the configuration block, instead of placing it in cleartext
Contributor guide
Research direction
Issue #4198 names no source file, test, or entry point, so begin by locating Fluentd configuration handling and the Elasticsearch output plugin's credential processing. Done means the Elasticsearch password is not stored in cleartext in the configuration while authentication continues to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, ruby
- Domain
- backend, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100