Config File - Parse steps from a config file
- Dominant language
- C
- Stars
- 4
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
It's been noted that passing steps on the command line isn't the most user friendly way of configuring the pipeline. Because it's required to fit all options into a single string, and the most obvious way to do that so far has been with URL style encoded parameters, it means that the ease of providing complex arguments to plugins (such as a SQL query, or a regular expression) is an absolute PITA.
However being able to construct simple pipelines quickly on the CLI means it's very easy to get started and experiment, then possibly add some simple options to tweak the behaviour.
But this model doesn't scale, and because the general aim is to provide an enabling framework for more complex logging pipelines to be written with it, we must provide a way to pass more complex configuration options to steps in the pipeline.
The proposal is to provide a configuration file format that can be encoded into the current URL style strings, this means there should be no module API changes to introduce the feature.
Initially I think that a Windows INI file format would be easy to achieve.
Example simple config:
```
[stdin]
[stdout]
```
or
```
[stdin]
[parse.apacheclf]
[syslog]
facility=INFO
[stdout]
```
A parser will be easy to construct with Ragel to avoid using an external library.
One edge-case I can think of is what happens if you provide both a configuration file and additional arguments on the command line? I think that the arguments on the command line should be added after the other steps, this allows for complex beginnings of pipelines to be extended with additional steps.
Additionally the configuration parsing should be provided as a module that can inject steps into the pipeline, e.g.:
```
logpipe stdin config:middle.ini stdout
```
This allows for modularity of complex sequences of steps to enable faster experimentation and re-use.
Being able to pass `config:middle.ini` on the CLI means that I probably don't have to introduce getopt style parsing immediately, but it will probably be necessary for #7.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.