getsentry / getsentry/sentry

`--create-kafka-topics` does not create topics

Open
#103,438 6 comments 0 reactions 0 assignees View on GitHub
Bug Self-hosted Waiting for: Product Owner
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
22h 21m
Merged PRs (30d)
586

Description

### Self-Hosted Version

25.7.0

### CPU Architecture

x86_64

### Docker Version

20.10.17

### Docker Compose Version

none

### Machine Specification

- [x] My system meets the minimum system requirements of Sentry

### Steps to Reproduce

1. Install 25.6.2
1. Install 25.7.0
1. run `docker exec -it web sentry upgrade -v 9 --create-kafka-topics`

### Expected Result

Topics created automatically

### Actual Result

Topics only checked for existence. Did I miss something?
It was like this since introduction of `--create-kafka-topics` flag.

Thank you very much btw for it. It feels is so much better to know what topics are actually need to be created.
```
{"name":"sentry.batching-kafka-consumer","sentry.trace.trace_id":null,"event":"Topic 'taskworker-buffer' or its partitions are not ready, retrying...","level":"warning"}
Traceback (most recent call last):
File "/.venv/bin/sentry", line 4, in
raise SystemExit(main())
~~~~^^
File "/usr/src/sentry/src/sentry/runner/main.py", line 145, in main
func(**kwargs)
~~~~^^^^^^^^^^
File "/.venv/lib/python3.13/site-packages/click/core.py", line 1442, in __call__
return self.main(*args, **kwargs)
~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.13/site-packages/click/core.py", line 1363, in main
rv = self.invoke(ctx)
File "/.venv/lib/python3.13/site-packages/click/core.py", line 1830, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/.venv/lib/python3.13/site-packages/click/core.py", line 1226, in invoke
return ctx.invoke(self.callback, **ctx.params)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.13/site-packages/click/core.py", line 794, in invoke
return callback(*args, **kwargs)
File "/.venv/lib/python3.13/site-packages/click/decorators.py", line 34, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/src/sentry/src/sentry/runner/decorators.py", line 34, in inner
return ctx.invoke(f, *args, **kwargs)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.13/site-packages/click/core.py", line 794, in invoke
return callback(*args, **kwargs)
File "/usr/src/sentry/src/sentry/runner/commands/upgrade.py", line 157, in upgrade
_upgrade(
~~~~~~~~^
not noinput,
^^^^^^^^^^^^
...<5 lines>...
create_kafka_topics,
^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/src/sentry/src/sentry/runner/commands/upgrade.py", line 93, in _upgrade
create_topics(topic_defn["cluster"], [topic_defn["real_topic_name"]])
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/sentry/src/sentry/utils/batching_kafka_consumer.py", line 54, in create_topics
wait_for_topics(admin_client, topics)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/sentry/src/sentry/utils/batching_kafka_consumer.py", line 22, in wait_for_topics
raise RuntimeError(
f"Timeout when waiting for Kafka topic '{topic}' to become available, last error: {last_error}"
)
RuntimeError: Timeout when waiting for Kafka topic 'taskworker-buffer' to become available, last error: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Broker: Unknown topic or partition"}
```

### Event ID

_No response_

Contributor guide

Open the contributing guide

Research direction

The upgrade path is in src/sentry/runner/commands/upgrade.py, and topic creation and waiting are in src/sentry/utils/batching_kafka_consumer.py. Start by tracing _upgrade with --create-kafka-topics and reproduce the reported upgrade path. Done means the flag creates missing topics and the upgrade flow no longer times out when a topic is absent.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kafka, python
Domain
cli, distributed-systems, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.