fluent / fluent/fluent-operator

help request: Loki instance external to cluster returning 404

Open
#1,344 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
682
Forks
328
Avg merge
2d 23h
Merged PRs (30d)
13

Description

### Describe the issue

Hi,
I am trying to forward logs to a Loki instance outside of the Kubernetes cluster. I get the below 404 error in the fluent-bit container logs.
`[error] [output:loki:loki.0] logging-dev.example.com:443, HTTP status=404`
`404 page not found`

A default Loki install returns a 404 at logging-dev.example.com

I have used curl to access the Loki API from the fluent-bit debug container successfully.

I noticed the uri parameter doesn't exist when i export the `fluent-bit-config` secret and it isn't defined in [fluentbit-output-loki.yaml](https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/templates/fluentbit-output-loki.yaml) . Could this be part of the issue?

### How did you install fluent operator?

helm install -f custom-values.yml fluent-operator --create-namespace -n fluent-operator charts/fluent-operator/

custom-values.yml
---
containerRuntime: containerd

operator:

initcontainer:

repository: "docker"

tag: "27.2.1"

fluentbit:

crdsEnable: true

enable: true

image:

repository: "ghcr.io/fluent/fluent-operator/fluent-bit"

tag: "3.1.7-debug"

tls:

verify: false

input:

tail:

enable: true

refreshIntervalSeconds: 10

memBufLimit: 100MB

bufferMaxSize: ""

path: "/var/log/containers/*.log"

skipLongLines: true

readFromHead: false

storageType: memory

pauseOnChunksOverlimit: "off"

systemd:

enable: true

systemdFilter:

enable: true

filters: []

path: "/var/log/journal"

includeKubelet: true

stripUnderscores: "off"

storageType: memory

pauseOnChunksOverlimit: "off"

output:

loki:

enable: true

host: logging-dev.example.com

port: 443

uri: /loki/api/v1/push

httpUser: user

httpPassword: pass
---

### Additional context

Output of `fluent-bit-config` secret:
[Service]

Http_Server true

Parsers_File /fluent-bit/etc/parsers.conf

Parsers_File /fluent-bit/config/parsers_multiline.conf

[Input]

Name systemd

Path /var/log/journal

DB /fluent-bit/tail/systemd.db

DB.Sync Normal

Tag service.*

Systemd_Filter _SYSTEMD_UNIT=containerd.service

Systemd_Filter _SYSTEMD_UNIT=kubelet.service

Strip_Underscores off

storage.type memory

[Input]

Name tail

Path /var/log/containers/*.log

Read_from_Head false

Refresh_Interval 10

Skip_Long_Lines true

DB /fluent-bit/tail/pos.db

DB.Sync Normal

Mem_Buf_Limit 100MB

Parser cri

Tag kube.*

storage.type memory

[Filter]

Name lua

Match kube.*

script /fluent-bit/config/containerd.lua

call containerd

time_as_table true

[Filter]

Name kubernetes

Match kube.*

Kube_URL https://kubernetes.default.svc:443

Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token

Labels false

Annotations false

[Filter]

Name nest

Match kube.*

Operation lift

Nested_under kubernetes

Add_prefix kubernetes_

[Filter]

Name modify

Match kube.*

Remove stream

Remove kubernetes_pod_id

Remove kubernetes_host

Remove kubernetes_container_hash

[Filter]

Name nest

Match kube.*

Operation nest

Wildcard kubernetes_*

Nest_under kubernetes

Remove_prefix kubernetes_

[Filter]

Name lua

Match service.*

script /fluent-bit/config/systemd.lua

call add_time

time_as_table true

[Output]

Name loki

Match_Regex (?:kube|service)\.(.*)

host logging-dev.example.com

port 443

http_user user

http_passwd pass

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.