[EKS] [request]: Want to match log records by k8s annotation: `rewrite_tag` filter don't work after `kubernetes` filter in Fargate embedded FluentBit logging

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
I want to use Kubernetes Pod annotations for routing/identifying destination log group in CloudWatch Logs.

**Which service(s) is this request for?**
Fargate, EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**

I want to change log routing destination by Pods annotation.

For example, I annotate Pods for web with app=web, and also do it for NodeJS application with app=node-app.

Then I tried following configuration.

```
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-logging
namespace: aws-observability
data:
flb_log_cw: "true"

output.conf: |
[OUTPUT]
Name cloudwatch_logs
Match kube.*
region ap-northeast-1
log_group_name fluent-bit-cloudwatch
log_stream_prefix from-fluent-bit-on-fargate-
auto_create_group true
[OUTPUT]
Name cloudwatch_logs
Match log.to.app-log-web.group.*
region ap-northeast-1
log_group_name app-log-web
log_stream_prefix from-fluent-bit-on-fargate-
auto_create_group true
[OUTPUT]
Name cloudwatch_logs
Match log.to.app-log-node-app.group.*
region ap-northeast-1
log_group_name app-log-node-app
log_stream_prefix from-fluent-bit-on-fargate-
auto_create_group true

parsers.conf: |
[PARSER]
Name crio
Format Regex
Regex ^(?[^ ]+) (?stdout|stderr) (?P|F) (?.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z

filters.conf: |
[FILTER]
Name parser
Match kube.*
Key_name log
Parser crio
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Buffer_Size 0
Kube_Meta_Cache_TTL 300s
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name rewrite_tag
Match kube.*
Rule $stream ^(std.+)$ log.to.app-log-$kubernetes['annotations']['app'].group.$TAG false
```

But above configuration and any other configuration with `rewrite_tag` not works.

In actual, if I commented out last `[FILTER]` section, the fargate logging works well, and logs of fluent-bit itself sent to `-fluent-bit-logs` log group. But add `[FILTER]` section, the fluent-bit log group not get log stream for restarted Pod.

**Are you currently working around this issue?**
Try to use Fluent Bit side car.

**Additional context**

If I use similar configuration on managed NodeGroup and Fluent Bit run as DaemonSet, it works as expected.

**Attachments**
Nothing special.

Start by reproducing the supplied ConfigMap in EKS Fargate, focusing on the parser, kubernetes, and rewrite_tag filters and their CloudWatch Logs outputs. Compare the behavior with Fluent Bit running as a DaemonSet on a managed node group. Done means rewrite_tag can match the Kubernetes annotation after the kubernetes filter and route records to the intended log groups.

Written by the indexing model from the issue text.