elastic / elastic/integrations
Needed improvements around AWS log collection configuration issues
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 182
Description
We have received multiple reports related to AWS logs input settings being wrong or misleading to customers.
This relates to the error "Error creating runner from config: log_group_arn, log_group_name and log_group_name_prefix config parametercannot all be empty accessing config" that users can get when using the AWS Cloudwatch input through an AWS data stream.
We output this error if all settings are empty, which I was expecting to mean "none of them has not been filled".
We discovered though that due to how the [agent template](https://github.com/elastic/integrations/blob/4c435e02877dc2c8d8caa880a78cd0592e4533c6/packages/aws/data_stream/cloudwatch_logs/agent/stream/aws-cloudwatch.yml.hbs) (`aws.cloudwatch_logs` data stream) is written if all three settings are empty the template will be empty too. This was unexpected and is a bug.
These are the affected data streams:
- cloudtrail
- cloudwatch_logs
- ec2_logs
- elb_logs
- firewall_logs
- route53_public_logs
- route53_resolver_logs
- vpcflow
- waf
Some more details, related to AWS Cloudwatch filebeat input (var names are not go vars, they logically represent the user facing settings).
- config validation checks:
- if either of the 3 settings is empty [here](https://github.com/elastic/beats/blob/c8578a0f674f5cde47e3c9b021a5800d46abc170/x-pack/filebeat/input/awscloudwatch/config.go#L51)
- if log_group_name and log_group_name_prefix are both provided [here](https://github.com/elastic/beats/blob/c8578a0f674f5cde47e3c9b021a5800d46abc170/x-pack/filebeat/input/awscloudwatch/config.go#L56)
- if log_group_name or log_group_name_prefix are used without providing region [here](https://github.com/elastic/beats/blob/c8578a0f674f5cde47e3c9b021a5800d46abc170/x-pack/filebeat/input/awscloudwatch/config.go#L60)
- when creating a new input:
- log_group_arn is used to extract log_group_name and region and silently overrides log_group_name [here](https://github.com/elastic/beats/blob/c8578a0f674f5cde47e3c9b021a5800d46abc170/x-pack/filebeat/input/awscloudwatch/input.go#L72)
- region is set by default to us-east-1 if nothing is provided [here](https://github.com/elastic/beats/blob/c8578a0f674f5cde47e3c9b021a5800d46abc170/x-pack/libbeat/common/aws/credentials.go#LL55C34-L55C34)
- during input run phase:
- log_group_name and log_group_name_prefix are used to collect all cloudwatch group names to poll [here](https://github.com/elastic/beats/blob/c8578a0f674f5cde47e3c9b021a5800d46abc170/x-pack/filebeat/input/awscloudwatch/input.go#L129)
- if log_group_name_prefix is empty, []string{log_group_name} is used [here](https://github.com/elastic/beats/blob/c8578a0f674f5cde47e3c9b021a5800d46abc170/x-pack/filebeat/input/awscloudwatch/input.go#L237)
- otherwise DescribeLogGroups is used to collect all log groups with prefix matching filter, silently overriding log_group_name [here](https://github.com/elastic/beats/blob/c8578a0f674f5cde47e3c9b021a5800d46abc170/x-pack/filebeat/input/awscloudwatch/input.go#LL247C28-L247C50)
My conclusions, valid configurations are:
- log_group_arn; region from arn overrides region
- log_group_name and region (required by AWS SDK, not required in UI, silently defaults to us-east-1 - documented behaviour)
- log_group_name_prefix and region (required by AWS SDK, not required in UI, silently defaults to us-east-1 - documented behaviour)
Some more context on this from @aspacca:
> [...]the three settings are used to drive the log group(s) you want to ingest[...]
> Basically you can either subscribe to a single log group or multiple of them.
> For multiple log groups you use the log_group_name_prefix.
> For a single one you have the option to either provide the whole log_group_arn or the log_group_name.
> A note about the region:
> When you don't have the whole ARN you miss the region information, that must be provided.
> The region and the default_region are two different settings.
> Not every filebeat inputs/metricbeat modules provide a region setting, in such case we use the default_region, that's generic to AWS and not specific to an input or module, with a final fallback on "us-east-1".
> We usually use the default_region for making the initial call with the AWS SDK in order to collect information about the proper region we need for a specific resource.
> Beware that filebeat inputs/metricbeat modules can then infer/request the proper region to pass to the AWS SDK, that might or not might the region provided by the users: this is intended since we'd like to be resilient enough to 1. avoid the users to provide the region if we have other way to figure it on our own, 2. avoid failing if we know the region we figured out is the correct one, while the one provided by the users might not.
The proposal to address this is to:
- refactor the code to only allow the 3 aforementioned cases explicitly
- add logging for rejection cases
- add logging when fallbacks are being used
Contributor guide
Assessment
This issue has not been assessed yet.