redpanda-data / redpanda-data/connect

Refactor drop_on output to use the new APIs and support batching in the underlying output

Open
#1,798 0 comments 0 reactions 1 assignee View on GitHub

@Jeffail is already working on this.

Since Mar 9, 2023.

bughancement outputs
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

Currently, the following config will block forever:

input:
  generate:
    count: 10
    mapping: root = count("test")
    interval: 250ms

output:
  drop_on:
    back_pressure: 0s
    error: false
    output:
      sql_raw:
        driver: sqlite
        dsn: file::memory:?cache=shared
        query: SELECT 1
        batching:
          count: 2 # Change this to 1 to unblock
          processors:
            - log:
                message: ${! batch_index() } - ${! batch_size() }

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.