fluent / fluent/fluentd-docs-gitbook

Improvement: Documentation for multi process worker could be improved to explain the behavior for hybrid config (defined inside worker directive and outside as well)

Open
#326 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
CSS
Stars
46
Forks
136
Avg merge
4h 47m
Merged PRs (30d)
16

Description

I was trying out multi process worker for one of my use case where few pipeline need to run in dedicated worker while some pipeline need to be defined outside any worker directive and should run on any of the worker process. While doing this, I wasn't sure how the configuration were included/loaded in each worker process and was unclear on the behavior observed. After doing some POC, I could understand the behavior. I suggest adding this in the documentation sot that it might help others understand the working behavior quickly.

The POC was done using "http" plugin which supports multi process worker and created pipelines to use it in dedicated workers as well as running in all the workers. The messages were sent using curl command on different ports to see the output.

**Observations from POC:**

_Message forwarding behavior:_
o For same port exposed in different workers, messages are sent randomly to one of the workers exposing that port.
o For same port exposed in worker(s) and at global (outside any worker), messages are sent randomly to one of the workers from entire worker list (irrespective of the worker exposing that port or not)
o For unique ports exposed in different workers, messages are sent to specific worker exposing respective ports

_Filtering/Matching behavior:_
o Common config (input/filters/match) which is not defined in any worker is included/considered in each worker.
o Common configs(input/filters/match) are included prior to any other config defined under specific worker tag

Following diagram depicts the actual setup used for POC.
![image](https://user-images.githubusercontent.com/21258768/120789702-93576a80-c54f-11eb-8bf9-f681cd96b109.png)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.