fluent / fluent/fluent-operator
help request: Loki instance external to cluster returning 404
- Dominant language
- Go
- Stars
- 682
- Forks
- 328
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 13
Description
### Describe the issue
Hi,
I am trying to forward logs to a Loki instance outside of the Kubernetes cluster. I get the below 404 error in the fluent-bit container logs.
`[error] [output:loki:loki.0] logging-dev.example.com:443, HTTP status=404`
`404 page not found`
A default Loki install returns a 404 at logging-dev.example.com
I have used curl to access the Loki API from the fluent-bit debug container successfully.
I noticed the uri parameter doesn't exist when i export the `fluent-bit-config` secret and it isn't defined in [fluentbit-output-loki.yaml](https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-output-loki.yaml) . Could this be part of the issue?
### How did you install fluent operator?
helm install -f custom-values.yml fluent-operator --create-namespace -n fluent-operator charts/fluent-operator/
custom-values.yml
---
containerRuntime: containerd
operator:
initcontainer:
repository: "docker"
tag: "27.2.1"
fluentbit:
crdsEnable: true
enable: true
image:
repository: "ghcr.io/fluent/fluent-operator/fluent-bit"
tag: "3.1.7-debug"
tls:
verify: false
input:
tail:
enable: true
refreshIntervalSeconds: 10
memBufLimit: 100MB
bufferMaxSize: ""
path: "/var/log/containers/*.log"
skipLongLines: true
readFromHead: false
storageType: memory
pauseOnChunksOverlimit: "off"
systemd:
enable: true
systemdFilter:
enable: true
filters: []
path: "/var/log/journal"
includeKubelet: true
stripUnderscores: "off"
storageType: memory
pauseOnChunksOverlimit: "off"
output:
loki:
enable: true
host: logging-dev.example.com
port: 443
uri: /loki/api/v1/push
httpUser: user
httpPassword: pass
---
### Additional context
Output of `fluent-bit-config` secret:
[Service]
Http_Server true
Parsers_File /fluent-bit/etc/parsers.conf
Parsers_File /fluent-bit/config/parsers_multiline.conf
[Input]
Name systemd
Path /var/log/journal
DB /fluent-bit/tail/systemd.db
DB.Sync Normal
Tag service.*
Systemd_Filter _SYSTEMD_UNIT=containerd.service
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Strip_Underscores off
storage.type memory
[Input]
Name tail
Path /var/log/containers/*.log
Read_from_Head false
Refresh_Interval 10
Skip_Long_Lines true
DB /fluent-bit/tail/pos.db
DB.Sync Normal
Mem_Buf_Limit 100MB
Parser cri
Tag kube.*
storage.type memory
[Filter]
Name lua
Match kube.*
script /fluent-bit/config/containerd.lua
call containerd
time_as_table true
[Filter]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Labels false
Annotations false
[Filter]
Name nest
Match kube.*
Operation lift
Nested_under kubernetes
Add_prefix kubernetes_
[Filter]
Name modify
Match kube.*
Remove stream
Remove kubernetes_pod_id
Remove kubernetes_host
Remove kubernetes_container_hash
[Filter]
Name nest
Match kube.*
Operation nest
Wildcard kubernetes_*
Nest_under kubernetes
Remove_prefix kubernetes_
[Filter]
Name lua
Match service.*
script /fluent-bit/config/systemd.lua
call add_time
time_as_table true
[Output]
Name loki
Match_Regex (?:kube|service)\.(.*)
host logging-dev.example.com
port 443
http_user user
http_passwd pass
Contributor guide
Assessment
This issue has not been assessed yet.