redpanda-data / redpanda-data/connect

Deleting a stream via the REST API can block forever

Open
#2,742 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ash pls needs investigation streams
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Steps to reproduce:

  1. Start Connect:
> redpanda-connect streams
  1. Create a simple stream via the REST API with a failing output which blocks messages forever:
> curl http://localhost:4195/streams/bar -X POST --data-binary @- <<EOF
input:
  broker:
    inputs:
    - generate:
        count: 1
        interval: 1s
        mapping: root = ""

output:
  retry:
    output:
      http_client:
        url: localhost:8080
EOF
  1. Try to delete the stream via the REST API:
> curl http://localhost:4195/streams/bar -X DELETE

Note that the curl command will hang and cancelling it via Ctrl+C will produce the following in the Connect log:

WARN Failed to send message: localhost:8080: Post "localhost:8080": unsupported protocol scheme "localhost"  @service=benthos label="" path=root.output stream=bar
ERRO Encountered error whilst attempting to shut down gracefully: context canceled  @service=benthos stream=bar
INFO Some components prevented forced termination as they were either blocked from delivering data or from acknowledging delivered data within the shutdown timeout. This could potentially cause duplicate messages to be delivered on the next run.  @service=benthos stream=bar
ERRO Streams CRUD Error: context canceled          @service=benthos

No more logs are produced after this, but trying to recreate the stream returns an error message saying "Stream already exists". Running the DELETE command again succeeds and now the stream can be recreated.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the redpanda-connect streams command and the REST API POST and DELETE requests shown above, then trace the stream deletion and graceful shutdown path. Done means deleting a stream with a blocked failing output returns instead of hanging, the stream is no longer reported as existing, and it can be recreated successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.