quickwit-oss / quickwit-oss/quickwit
Getting a `failed to load IMDS session token` on a non-AWS environment
Open
Nobody has claimed this yet.
bug
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
observed on scaleway with the following config
version: 0.7
metastore_uri: s3://qw-data-xxxxxx
default_index_root_uri: s3://qw-data-xxxxxx
storage:
s3:
access_key_id: xxxxxxx
secret_access_key: yyyyyyyy
region: fr-par
endpoint: https://s3.fr-par.scw.cloud
force_path_style_access: ${QW_S3_FORCE_PATH_STYLE_ACCESS:-false}
disable_multi_object_delete: false
disable_multipart_upload: false
indexer:
enable_otlp_endpoint: ${QW_ENABLE_OTLP_ENDPOINT:-true}
jaeger:
enable_endpoint: ${QW_ENABLE_JAEGER_ENDPOINT:-true}
As reported by @ineumann
^[[2m2024-09-16T09:47:44.250Z^[[0m ^[[33m WARN^[[0m ^[[2maws_config::imds::region^[[0m^[[2m:^[[0m failed to load region from IMDS ^[[3merr^[[0m^[[2m=^[[0mfailed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
^[[2m2024-09-16T09:47:45.252Z^[[0m ^[[33m WARN^[[0m ^[[2maws_config::imds::region^[[0m^[[2m:^[[0m failed to load region from IMDS ^[[3merr^[[0m^[[2m=^[[0mfailed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
^[[2m2024-09-16T09:49:22.920Z^[[0m ^[[31mERROR^[[0m ^[[2mquickwit_proto::error^[[0m^[[2m:^[[0m gRPC transport error: Timeout expired ^[[3mcode^[[0m^[[2m=^[[0mCancelled ^[[3mrpc^[[0m^[[2m=^[[0m"publish_splits"
^[[2m2024-09-16T09:49:22.921Z^[[0m ^[[31mERROR^[[0m ^[[2mquickwit_actors::spawn_builder^[[0m^[[2m:^[[0m actor-failure ^[[3mcause^[[0m^[[2m=^[[0mfailed to publish splits
Could be similar to https://github.com/delta-io/delta-rs/pull/2817/files
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Scaleway configuration and tracing the AWS IMDS region lookup and S3 client initialization associated with the logged timeout. Determine why a non-AWS endpoint still attempts IMDS access, then verify that publishing splits succeeds without the repeated IMDS warnings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100