`--create-kafka-topics` does not create topics
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 22h 21m
- Merged PRs (30d)
- 586
Description
### Self-Hosted Version
25.7.0
### CPU Architecture
x86_64
### Docker Version
20.10.17
### Docker Compose Version
none
### Machine Specification
- [x] My system meets the minimum system requirements of Sentry
### Steps to Reproduce
1. Install 25.6.2
1. Install 25.7.0
1. run `docker exec -it web sentry upgrade -v 9 --create-kafka-topics`
### Expected Result
Topics created automatically
### Actual Result
Topics only checked for existence. Did I miss something?
It was like this since introduction of `--create-kafka-topics` flag.
Thank you very much btw for it. It feels is so much better to know what topics are actually need to be created.
```
{"name":"sentry.batching-kafka-consumer","sentry.trace.trace_id":null,"event":"Topic 'taskworker-buffer' or its partitions are not ready, retrying...","level":"warning"}
Traceback (most recent call last):
File "/.venv/bin/sentry", line 4, in
raise SystemExit(main())
~~~~^^
File "/usr/src/sentry/src/sentry/runner/main.py", line 145, in main
func(**kwargs)
~~~~^^^^^^^^^^
File "/.venv/lib/python3.13/site-packages/click/core.py", line 1442, in __call__
return self.main(*args, **kwargs)
~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.13/site-packages/click/core.py", line 1363, in main
rv = self.invoke(ctx)
File "/.venv/lib/python3.13/site-packages/click/core.py", line 1830, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/.venv/lib/python3.13/site-packages/click/core.py", line 1226, in invoke
return ctx.invoke(self.callback, **ctx.params)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.13/site-packages/click/core.py", line 794, in invoke
return callback(*args, **kwargs)
File "/.venv/lib/python3.13/site-packages/click/decorators.py", line 34, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/src/sentry/src/sentry/runner/decorators.py", line 34, in inner
return ctx.invoke(f, *args, **kwargs)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.13/site-packages/click/core.py", line 794, in invoke
return callback(*args, **kwargs)
File "/usr/src/sentry/src/sentry/runner/commands/upgrade.py", line 157, in upgrade
_upgrade(
~~~~~~~~^
not noinput,
^^^^^^^^^^^^
...<5 lines>...
create_kafka_topics,
^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/src/sentry/src/sentry/runner/commands/upgrade.py", line 93, in _upgrade
create_topics(topic_defn["cluster"], [topic_defn["real_topic_name"]])
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/sentry/src/sentry/utils/batching_kafka_consumer.py", line 54, in create_topics
wait_for_topics(admin_client, topics)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/sentry/src/sentry/utils/batching_kafka_consumer.py", line 22, in wait_for_topics
raise RuntimeError(
f"Timeout when waiting for Kafka topic '{topic}' to become available, last error: {last_error}"
)
RuntimeError: Timeout when waiting for Kafka topic 'taskworker-buffer' to become available, last error: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Broker: Unknown topic or partition"}
```
### Event ID
_No response_
Contributor guide
Research direction
The upgrade path is in src/sentry/runner/commands/upgrade.py, and topic creation and waiting are in src/sentry/utils/batching_kafka_consumer.py. Start by tracing _upgrade with --create-kafka-topics and reproduce the reported upgrade path. Done means the flag creates missing topics and the upgrade flow no longer times out when a topic is absent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kafka, python
- Domain
- cli, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100