quickwit-oss / quickwit-oss/quickwit

Getting a `failed to load IMDS session token` on a non-AWS environment

Open
#5,435 4 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

observed on scaleway with the following config

version: 0.7

metastore_uri: s3://qw-data-xxxxxx
default_index_root_uri: s3://qw-data-xxxxxx

storage:
  s3:
    access_key_id: xxxxxxx
    secret_access_key: yyyyyyyy
    region: fr-par
    endpoint: https://s3.fr-par.scw.cloud
    force_path_style_access: ${QW_S3_FORCE_PATH_STYLE_ACCESS:-false}
    disable_multi_object_delete: false
    disable_multipart_upload: false

indexer:
  enable_otlp_endpoint: ${QW_ENABLE_OTLP_ENDPOINT:-true}

jaeger:
  enable_endpoint: ${QW_ENABLE_JAEGER_ENDPOINT:-true}

As reported by @ineumann

^[[2m2024-09-16T09:47:44.250Z^[[0m ^[[33m WARN^[[0m ^[[2maws_config::imds::region^[[0m^[[2m:^[[0m failed to load region from IMDS ^[[3merr^[[0m^[[2m=^[[0mfailed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
^[[2m2024-09-16T09:47:45.252Z^[[0m ^[[33m WARN^[[0m ^[[2maws_config::imds::region^[[0m^[[2m:^[[0m failed to load region from IMDS ^[[3merr^[[0m^[[2m=^[[0mfailed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
^[[2m2024-09-16T09:49:22.920Z^[[0m ^[[31mERROR^[[0m ^[[2mquickwit_proto::error^[[0m^[[2m:^[[0m gRPC transport error: Timeout expired ^[[3mcode^[[0m^[[2m=^[[0mCancelled ^[[3mrpc^[[0m^[[2m=^[[0m"publish_splits"
^[[2m2024-09-16T09:49:22.921Z^[[0m ^[[31mERROR^[[0m ^[[2mquickwit_actors::spawn_builder^[[0m^[[2m:^[[0m actor-failure ^[[3mcause^[[0m^[[2m=^[[0mfailed to publish splits

Could be similar to https://github.com/delta-io/delta-rs/pull/2817/files

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Scaleway configuration and tracing the AWS IMDS region lookup and S3 client initialization associated with the logged timeout. Determine why a non-AWS endpoint still attempts IMDS access, then verify that publishing splits succeeds without the repeated IMDS warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.