fluent / fluent/fluent-plugin-mongo

How to transfer data to Amazon DocumentDB

Open
#160 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
175
Forks
56
PR merge metrics
No merged PRs in 30d

Description

I can use Fluentd to transfer data to mongoDB built on AWS EC2, but I can't transfer data to DocumentDB, which is a managed service compatible with mongoDB.

The following is the **td-agent.conf** for transferring the json file saved in **/var/log/test/bulk/** to mongoDB.

```

@type tail
path /var/log/test/bulk/*
tag bulk.*
format json
time_key time
time_format '%F %T.%N %z %Z'
pos_file /var/log/test/run/log-json.pos
read_from_head true
refresh_interval 5s

@type record_reformer
tag test.${tag_parts[-3]}.${tag_parts[-2]}

@type copy

@type forest
subtype mongo_replset

nodes hostname1:27017,hostname2:27017,hostname3:27017
replica_set rs0
database ${tag_parts[-2]}
collection ${tag_parts[-1]}
user ********
password ********
replace_dot_in_key_with __dot__

@type file
path /var/log/test/buffer-mongo/${tag_parts[-2..-1]}
chunk_limit_size 8m
queued_chunks_limit_size 64
flush_interval 1s


```

When transferring to DocumentDB, I changed the host in the conf file above to the cluster endpoint, but the following error occurred.

```
[warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-14 10:26:56 +0900 chunk="5cbea78155b58ec0810e9fde94aa2355" error_class=Mongo::Error::NoServerAvailable error="No server is available matching preference: # using server_selection_timeout=30 and local_threshold=0.015"
```

Since TLS is enabled in DocumentDB, I wonder if I need to specify [rds-combined-ca-bundle.pem](https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem) to enable TLS. I think so, but I don't know how to do that. (When I tested writing to DocumentDB in Python using [the link](https://docs.aws.amazon.com/ja_jp/documentdb/latest/developerguide/connect_programmatically.html), the above error occurred when TLS was disabled.)

Is it possible to write data to DocumentDB with TLS enabled?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.