Upgrade Vector to version 0.58.0

previous ticket: #819

Version 0.58.0

https://vector.dev/releases/0.58.0/

No changes are required.

Version 0.57.0

https://vector.dev/releases/0.57.0/

Environment variable interpolation in configuration files is now disabled by default. Previously, Vector automatically interpolated ${VAR} references found in config files.
[...]

Environment-variable and secret placeholders in non-string positions

Placeholders (${VAR} and SECRET[backend.key]) in structural positions of a Vector configuration file are deprecated and will be removed in a future release. Current behavior is unchanged.

Affected patterns:

• Unquoted placeholders in non-string fields (count = ${MY_COUNT}).
• Placeholders in map keys (${KEY} = value).
• Placeholders in TOML table headers ([${SECTION}]).
• Placeholders as inline array elements that expand to multiple values (inputs = [${VECTOR_INPUTS}] where VECTOR_INPUTS=a,b).

Environment variables are used extensively, e.g. to set the namespace, cluster name, role name, role group name, file locations and so on. However, the environment variables are in "string positions" and not in "structural positions".

Add support for configuring multiple endpoints in the vector sink via the new routing.endpoints option, enabling built-in load_balance, failover, and failover_primary endpoint strategies across downstream Vector instances. The previous address option is now deprecated in favor of routing.endpoints.

The address option should be replaced with routing.endpoints.

The vector sink now supports optional HTTP/2 keepalive on its pooled gRPC connections, configured via a new keepalive block (interval_secs and timeout_secs). When enabled, the sink periodically sends HTTP/2 PING frames so that a connection to a downstream Vector instance that has gone away is detected and evicted rather than reused indefinitely (which could otherwise stall delivery until the connection was replaced). Keepalive is disabled by default; when enabled, interval_secs defaults to 60 (aligned with gRPC keepalive guidance to avoid tripping too_many_pings policies) and timeout_secs defaults to 20. PINGs are sent on idle connections so an idle-but-dead connection is still detected.

It could be useful to enable the keepalive feature.

Version 0.56.0

https://vector.dev/releases/0.56.0/

The vector sink now supports zstd compression in addition to gzip. This provides better compression ratios and performance for Vector-to-Vector communication.

We do not use compression for the communication from the Vector agent to the Vector aggregator (https://github.com/stackabletech/operator-rs/blob/stackable-operator-0.111.1/crates/stackable-operator/src/product_logging/framework.rs#L1307-L1311), but we should.

Unit tests now support an optional expected_event_count field on test outputs, allowing assertions on the number of events emitted by a transform.

Only the OpenSearch operator has unit tests for Vector (https://github.com/stackabletech/opensearch-operator/blob/26.3.0/rust/operator-binary/src/controller/build/product_logging/vector-test.yaml). The new field is not required there, but could be useful in future unit tests.

Fixed a bug in the file source where checkpoints recording the last-read file position were not always fully written before Vector shut down. On the next startup, the file source could start reading from an earlier position, causing events to be re-processed.

This is the corresponding pull request: vectordotdev/vector#25447

We only encountered duplicated logs in stackabletech/issues#821, but this fix is completely unrelated.

Start with previous ticket #819 and the Vector 0.58.0 release notes. Review framework.rs around the Vector agent-to-aggregator configuration and the OpenSearch Vector test configuration linked in the issue, then check where the Vector sink address and compression are configured. Done means the relevant repositories use Vector 0.58.0 and the applicable release changes are addressed or explicitly ruled out.