redpanda-data / redpanda-data/connect

GCP cloud storage output error with Failed to delete temporary file used for merging: context canceled

Open
#2,522 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bughancement needs investigation
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Context

I am reading bunch of event coming from a MQTT broker

input:
  label: "benthos_mqtt_test"
  mqtt:
    urls:
      - localhost:1883
    client_id: benthos-reader
    connect_timeout: 30s
    topics:
      - $share/benthos/XXXX/#

and I am trying to process and save the output into blobstorage (GCP cloud storage).
The processing is relatively simple, I am taking an mqtt message (slice of events) and outputting to a CSV like format

buffer:
  none: {}
pipeline:
  threads: -1
  processors: 
    - unarchive:
        format: "json_map"
    - bloblang: |
        let csv = match {
             this.value.type() == "bool" => ",,"+this.value.string()
             this.value.type() == "number" => this.value.string()+",,"
             _ =>  ","+this.value.string()+","
        }
        root = this.ts.ts_strftime("%Y-%m-%dT%H:%M:%S%z").string() + "," + meta("archive_key") + "," + $csv +"\n"
        meta dt = this.ts.ts_strftime("%Y-%m-%d", "UTC")
        meta id = meta("archive_key").escape_url_query()

the output logic is to create folder partition by date (dt=YYYY/MM/DD) and id (id=ID)

output:
  label: "blobstorage"
  #stdout: {}
  gcp_cloud_storage:
    bucket: "abucket" 
    path: benthos/dt=${!@dt}/id=${!@id}/file.csv
    collision_mode: append
    timeout: 60s
    max_in_flight: 64

when running Benthos with the config, I am seeing quite often this error

ERRO Failed to delete temporary file used for merging: context canceled  @service=benthos label=blobstorage path=root.output

I am not 100% sure if this is false positive since I have quite a lot of event comming per sec (~1000 events) and the error doesn't tell me much about the path timing out so I cannot really check if the temp file has been merged or not.
i tried to change timeout 20s, 30s 60s but seems to always error.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the gcp_cloud_storage output configuration and its collision_mode: append behavior, using the supplied MQTT pipeline and high-throughput settings to reproduce the context-canceled deletion error. Check whether the temporary file was merged despite the error and identify which timeout or cancellation path produces the message. Done means documenting the cause and making the error outcome unambiguous.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.