vectordotdev / vectordotdev/vector
Expand `vector generate` pipeline syntax to accept arguments
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
Something that came up recently, specifically for our guides, is the ability to start Vector without passing a vector.toml file. This would be useful for very simple pipelines and examples. We already have a short-style syntax for vector generate:
vector generate 'stdin | aws_s3`
The config generated from this command requires manual modification because required arguments are not set, like the S3 bucket. What if the generate syntax could accept arguments? Then this would work:
vector $(vector generate 'stdin | aws_s3 --region="us-west-1" --bucket="my_bucket" --request.batch_size=10000000')
And if that works, we might as well support inline configs as an argument:
vector 'stdin | aws_s3 --region="us-west-1" --bucket="my_bucket"' --request.batch_size=10000000
I'm curious what people think about the possible syntaxes:
- Bash style / command line syntax (my favorite)
- A URL syntax like Apache Camel uses.
- Any others?
To be clear, this is for simple linear pipelines and should not support complex cases. The vector.toml syntax is designed for more complex pipelines.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing vector generate short-style pipeline syntax and the CLI entry points that parse it. Compare the proposed Bash-style arguments, URL syntax, and inline configuration, then define the supported scope for simple linear pipelines and verify the examples in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100