Azure-Samples / Azure-Samples/azure-sql-db-change-stream-debezium

history eventhub is configured with --cleanup-policy Delete instead of Compact.

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
105
Forks
37
PR merge metrics
No merged PRs in 30d

Description

The sample history hub configured is configured as follows.

```
echo "deploying schema history event hub"
az eventhubs eventhub create \
--resource-group $RESOURCE_GROUP \
--namespace-name $EVENTHUB_NAMESPACE \
--name $EVENTHUB_SCHEMA_HISTORY \
--partition-count 1 \
--cleanup-policy Delete \
--retention-time-in-hours 168 \
--output none
```

This seems like terrible advice for any production environment.
Shouldn't it be cleanup-policy compact and infinite retention?

however, when I set this I cannot get debezium to write to the topic/hub without CORRUPT_MESSAGE errors. Works fine when --cleanup-policy Delete

Contributor guide

Open the contributing guide

Research direction

Start with the Azure CLI Event Hubs creation command in the issue and compare the Delete configuration with Compact and the retention setting. Reproduce the Debezium write using each configuration and investigate the reported CORRUPT_MESSAGE errors; the work is done when the sample configuration or its documented limitation is clear and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, kafka
Domain
cloud, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.