vectordotdev / vectordotdev/vector

Map `source` field in `splunk_hec` sink

Open
#2,268 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

have: should needs: approval sink: splunk_hec_logs
Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

#2147 originally attempted to introduce a new source field across all of Vector's sources. This is a Splunk concept that attempts to abstract source information into a single field. This was intended to replace specific context keys within the source like file, container_name, topic_name, and so on. While I can see the value in this field, it starts to get awkward when we attempt to shoehorn multiple pieces of context into a single field. A better solution is to allow each source to set and define it's own context keys and then perform the source field mapping within the splunk_hec sink, where it is relevant. This is similar to how we map other fields in other sinks.

To provide a concrete list of mappings:

  • When source_type is docker set the source field to {{ container_name }}:{{ container_id }} (open to feedback here).
  • When source_type is file set the source field to {{ file }}.
  • When source_type is http set the source field to {{ address }}. Blocked by #2269.
  • When source_type is journald set the source field to {{_SYSTEMD_UNIT }}.
  • When source_type is kafka set the source field to {{ topic }}:{{ partition }}. Blocked by #1150.
  • When source_type is logplex set the source field to {{ address }}. Blocked by #2271.
  • When source_type is socket set the source field to {{ protocol }}:{{ address | peer_path }}. Blocked by #2270.
  • When source_type is splunk_hec set the source field to {{ address }}. Blocked by #2273.
  • When source_type is stdin set the source field to stdin.
  • When source_type is syslog set the source field to {{ protocol }}:{{ address | peer_path }}. Blocked by #2272.
  • When source_type is vector set the source field to {{ address }}.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the splunk_hec sink and review the source-type mapping checklist in this issue. Implement the unblocked mappings, account for the issues marked as blockers, and consider the work done when the relevant events receive the specified Splunk source value.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.