Redpanda connection issue

I'm running the docker compose config for huly in a fresh debian 13 LXC.

I set up my reverse proxy, made an account, created a workspace, which created, and now it loads forever, with this in the docker logs. (nothing clientside, in console or network tabs).

I've tried disabling the healthcheck on redpanda, no luck

```
transactor-1 | {"err":{"message":"Connection error: connect ECONNREFUSED 172.18.0.7:9092","name":"KafkaJSNumberOfRetriesExceeded","retriable":false,"retryCount":5,"retryTime":7434,"stack":"KafkaJSNonRetriableError\n Caused by: KafkaJSConnectionError: Connection error: connect ECONNREFUSED 172.18.0.7:9092\n at Socket.onError (/usr/src/app/bundle.js:137690:28)\n at Socket.emit (node:events:519:28)\n at emitErrorNT (node:internal/streams/destroy:170:8)\n at emitErrorCloseNT (node:internal/streams/destroy:129:3)\n at process.processTicksAndRejections (node:internal/process/task_queues:90:21)"},"level":"error","message":"unexpected error in websocket","timestamp":"2026-03-12T15:07:27.472Z"}
transactor-1 | {"level":"error","message":"Unhandled Rejection at:","promise":{},"reason":{"message":"Connection error: connect ECONNREFUSED 172.18.0.7:9092","name":"KafkaJSNumberOfRetriesExceeded","retriable":false,"retryCount":5,"retryTime":7434,"stack":"KafkaJSNonRetriableError\n Caused by: KafkaJSConnectionError: Connection error: connect ECONNREFUSED 172.18.0.7:9092\n at Socket.onError (/usr/src/app/bundle.js:137690:28)\n at Socket.emit (node:events:519:28)\n at emitErrorNT (node:internal/streams/destroy:170:8)\n at emitErrorCloseNT (node:internal/streams/destroy:129:3)\n at process.processTicksAndRejections (node:internal/process/task_queues:90:21)"},"timestamp":"2026-03-12T15:07:27.472Z"}
transactor-1 | (node:1) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 4)
transactor-1 | {
transactor-1 | message: 'Failed to process session operation',
transactor-1 | err: KafkaJSNonRetriableError
transactor-1 | Caused by: KafkaJSConnectionError: Connection error: connect ECONNREFUSED 172.18.0.7:9092
transactor-1 | at Socket.onError (/usr/src/app/bundle.js:137690:28)
transactor-1 | ... 3 lines matching cause stack trace ...
transactor-1 | at process.processTicksAndRejections (node:internal/process/task_queues:90:21) {
transactor-1 | name: 'KafkaJSNumberOfRetriesExceeded',
transactor-1 | retriable: false,
transactor-1 | helpUrl: undefined,
transactor-1 | retryCount: 5,
transactor-1 | retryTime: 7434,
transactor-1 | [cause]: KafkaJSConnectionError: Connection error: connect ECONNREFUSED 172.18.0.7:9092
transactor-1 | at Socket.onError (/usr/src/app/bundle.js:137690:28)
transactor-1 | at Socket.emit (node:events:519:28)
transactor-1 | at emitErrorNT (node:internal/streams/destroy:170:8)
transactor-1 | at emitErrorCloseNT (node:internal/streams/destroy:129:3)
transactor-1 | at process.processTicksAndRejections (node:internal/process/task_queues:90:21) {
transactor-1 | retriable: true,
transactor-1 | helpUrl: undefined,
transactor-1 | broker: 'redpanda:9092',
transactor-1 | code: 'ECONNREFUSED',
transactor-1 | [cause]: undefined
transactor-1 | }
transactor-1 | }
transactor-1 | }
```

Last redpanda logs:

```
...
redpanda-1 | INFO 2026-03-12 15:05:18,490 [shard 0:main] cluster - controller_backend.cc:833 - Cleaning up orphan topic files. bootstrap_revision: -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:18,494 [shard 0:main] cluster - feature_manager.cc:100 - Starting...
redpanda-1 | INFO 2026-03-12 15:05:18,494 [shard 0:main] cluster - feature_manager.cc:625 - Activating features after upgrade...
redpanda-1 | INFO 2026-03-12 15:05:18,495 [shard 0:main] cluster - feature_manager.cc:634 - Activating feature broker_time_based_retention (logical version 14)
redpanda-1 | INFO 2026-03-12 15:05:18,495 [shard 0:main] cluster - metrics_reporter.cc:342 - Waiting to initialize cluster metrics ID...
redpanda-1 | INFO 2026-03-12 15:05:18,497 [shard 0:main] cluster - partition_balancer_backend.cc:108 - partition balancer started
redpanda-1 | INFO 2026-03-12 15:05:18,498 [shard 0:main] data-migrate - data_migration_backend.cc:107 - backend starting
redpanda-1 | INFO 2026-03-12 15:05:18,498 [shard 0:main] data-migrate - data_migration_backend.cc:160 - backend not started as cloud_storage_api is not available
redpanda-1 | INFO 2026-03-12 15:05:18,500 [shard 0:main] cluster - leader_balancer.cc:112 - Leader balancer: controller leadership detected. Starting rebalancer in 30 seconds
redpanda-1 | INFO 2026-03-12 15:05:18,583 [shard 0:main] cluster - config_manager.cc:164 - Importing property auto_create_topics_enabled:1
redpanda-1 | INFO 2026-03-12 15:05:18,583 [shard 0:main] cluster - config_manager.cc:164 - Importing property storage_min_free_bytes:10485760
redpanda-1 | INFO 2026-03-12 15:05:18,583 [shard 0:main] cluster - config_manager.cc:164 - Importing property group_initial_rebalance_delay:0
redpanda-1 | INFO 2026-03-12 15:05:18,583 [shard 0:main] cluster - config_manager.cc:164 - Importing property write_caching_default:true
redpanda-1 | INFO 2026-03-12 15:05:18,583 [shard 0:main] cluster - config_manager.cc:164 - Importing property log_segment_size_min:{1}
redpanda-1 | INFO 2026-03-12 15:05:18,583 [shard 0:main] cluster - config_manager.cc:164 - Importing property fetch_reads_debounce_timeout:10
redpanda-1 | INFO 2026-03-12 15:05:18,583 [shard 0:main] cluster - config_manager.cc:164 - Importing property group_topic_partitions:3
redpanda-1 | INFO 2026-03-12 15:05:18,610 [shard 0:main] cluster - feature_backend.cc:149 - Saving feature_table_snapshot at version 14...
redpanda-1 | WARN 2026-03-12 15:05:18,639 [shard 0:main] admin_api_server - server.cc:547 - Insecure Admin API listener on 0.0.0.0:9644, consider enabling `admin_api_require_auth`
redpanda-1 | INFO 2026-03-12 15:05:18,639 [shard 0:main] admin_api_server - server.cc:353 - Started HTTP admin service listening at {{:{host: 0.0.0.0, port: 9644}}}
redpanda-1 | INFO 2026-03-12 15:05:18,639 [shard 0:main] resource_mgmt - storage.cc:73 - Starting disk space manager service (enabled)
redpanda-1 | INFO 2026-03-12 15:05:18,660 [shard 0:main] main - application.cc:2874 - Started Pandaproxy listening at {{internal:{host: 0.0.0.0, port: 8082}:}, {external:{host: 0.0.0.0, port: 18082}:}}
redpanda-1 | INFO 2026-03-12 15:05:18,670 [shard 0:main] cluster - metrics_reporter.cc:390 - Generated cluster metrics ID b18734af-4e0f-4226-aabd-cf2e20a2d416
redpanda-1 | INFO 2026-03-12 15:05:18,676 [shard 0:main] main - application.cc:2882 - Started Schema Registry listening at {{internal:{host: 0.0.0.0, port: 8081}:}, {external:{host: 0.0.0.0, port: 18081}:}}
redpanda-1 | INFO 2026-03-12 15:05:18,676 [shard 0:main] main - application.cc:3258 - Waiting for cluster membership
redpanda-1 | INFO 2026-03-12 15:05:18,676 [shard 0:main] main - application.cc:3276 - Waiting for Cluster ID to initialize...
redpanda-1 | INFO 2026-03-12 15:05:18,677 [shard 0:main] cluster - config_manager.cc:136 - Completed bootstrap as leader
redpanda-1 | INFO 2026-03-12 15:05:18,677 [shard 0:main] cluster - config_manager.cc:124 - Bootstrap complete (version 1)
redpanda-1 | INFO 2026-03-12 15:05:18,680 [shard 0:main] main - application.cc:3287 - Started Kafka API server listening at {{internal:{host: 0.0.0.0, port: 9092}:{nullopt}}, {external:{host: 0.0.0.0, port: 19092}:{nullopt}}}
redpanda-1 | INFO 2026-03-12 15:05:18,687 [shard 0:main] main - application.cc:2925 - Successfully started Redpanda!
redpanda-1 | INFO 2026-03-12 15:05:18,687 [shard 0:main] cluster - metrics_reporter.cc:420 - Initialized cluster_id to b18734af-4e0f-4226-aabd-cf2e20a2d416
redpanda-1 | INFO 2026-03-12 15:05:40,941 [shard 0:main] cluster - topics_frontend.cc:346 - Create topics [{configuration: { topic: {kafka/tx}, partition_count: 1, replication_factor: 1, is_migrated: false, properties: { compression: {nullopt}, cleanup_policy_bitflags: {nullopt}, compaction_strategy: {nullopt}, retention_bytes: {{nullopt}}, retention_duration_ms: {{nullopt}}, segment_size: {nullopt}, timestamp_type: {nullopt}, recovery_enabled: {nullopt}, shadow_indexing: {disabled}, read_replica: {nullopt}, read_replica_bucket: {nullopt}, remote_topic_namespace_override: {nullopt}, remote_topic_properties: {nullopt}, batch_max_bytes: {nullopt}, retention_local_target_bytes: {{nullopt}}, retention_local_target_ms: {{nullopt}}, remote_delete: true, segment_ms: {{nullopt}}, record_key_schema_id_validation: {nullopt}, record_key_schema_id_validation_compat: {nullopt}, record_key_subject_name_strategy: {nullopt}, record_key_subject_name_strategy_compat: {nullopt}, record_value_schema_id_validation: {nullopt}, record_value_schema_id_validation_compat: {nullopt}, record_value_subject_name_strategy: {nullopt}, record_value_subject_name_strategy_compat: {nullopt}, initial_retention_local_target_bytes: {{nullopt}}, initial_retention_local_target_ms: {{nullopt}}, mpx_virtual_cluster_id: {nullopt}, write_caching: {nullopt}, flush_ms: {nullopt}, flush_bytes: {nullopt}, remote_label: {nullopt}, iceberg_mode: disabled, leaders_preference: {nullopt}, delete_retention_ms: {disabled}, iceberg_delete: {nullopt}}}, custom_assignments: {}},]
redpanda-1 | INFO 2026-03-12 15:05:40,952 [shard 0:main] cluster - topics_frontend.cc:346 - Create topics [{configuration: { topic: {kafka/fulltext}, partition_count: 1, replication_factor: 1, is_migrated: false, properties: { compression: {nullopt}, cleanup_policy_bitflags: {nullopt}, compaction_strategy: {nullopt}, retention_bytes: {{nullopt}}, retention_duration_ms: {{nullopt}}, segment_size: {nullopt}, timestamp_type: {nullopt}, recovery_enabled: {nullopt}, shadow_indexing: {disabled}, read_replica: {nullopt}, read_replica_bucket: {nullopt}, remote_topic_namespace_override: {nullopt}, remote_topic_properties: {nullopt}, batch_max_bytes: {nullopt}, retention_local_target_bytes: {{nullopt}}, retention_local_target_ms: {{nullopt}}, remote_delete: true, segment_ms: {{nullopt}}, record_key_schema_id_validation: {nullopt}, record_key_schema_id_validation_compat: {nullopt}, record_key_subject_name_strategy: {nullopt}, record_key_subject_name_strategy_compat: {nullopt}, record_value_schema_id_validation: {nullopt}, record_value_schema_id_validation_compat: {nullopt}, record_value_subject_name_strategy: {nullopt}, record_value_subject_name_strategy_compat: {nullopt}, initial_retention_local_target_bytes: {{nullopt}}, initial_retention_local_target_ms: {{nullopt}}, mpx_virtual_cluster_id: {nullopt}, write_caching: {nullopt}, flush_ms: {nullopt}, flush_bytes: {nullopt}, remote_label: {nullopt}, iceberg_mode: disabled, leaders_preference: {nullopt}, delete_retention_ms: {disabled}, iceberg_delete: {nullopt}}}, custom_assignments: {}},]
redpanda-1 | INFO 2026-03-12 15:05:40,952 [shard 0:main] cluster - topics_frontend.cc:346 - Create topics [{configuration: { topic: {kafka/workspace}, partition_count: 1, replication_factor: 1, is_migrated: false, properties: { compression: {nullopt}, cleanup_policy_bitflags: {nullopt}, compaction_strategy: {nullopt}, retention_bytes: {{nullopt}}, retention_duration_ms: {{nullopt}}, segment_size: {nullopt}, timestamp_type: {nullopt}, recovery_enabled: {nullopt}, shadow_indexing: {disabled}, read_replica: {nullopt}, read_replica_bucket: {nullopt}, remote_topic_namespace_override: {nullopt}, remote_topic_properties: {nullopt}, batch_max_bytes: {nullopt}, retention_local_target_bytes: {{nullopt}}, retention_local_target_ms: {{nullopt}}, remote_delete: true, segment_ms: {{nullopt}}, record_key_schema_id_validation: {nullopt}, record_key_schema_id_validation_compat: {nullopt}, record_key_subject_name_strategy: {nullopt}, record_key_subject_name_strategy_compat: {nullopt}, record_value_schema_id_validation: {nullopt}, record_value_schema_id_validation_compat: {nullopt}, record_value_subject_name_strategy: {nullopt}, record_value_subject_name_strategy_compat: {nullopt}, initial_retention_local_target_bytes: {{nullopt}}, initial_retention_local_target_ms: {{nullopt}}, mpx_virtual_cluster_id: {nullopt}, write_caching: {nullopt}, flush_ms: {nullopt}, flush_bytes: {nullopt}, remote_label: {nullopt}, iceberg_mode: disabled, leaders_preference: {nullopt}, delete_retention_ms: {disabled}, iceberg_delete: {nullopt}}}, custom_assignments: {}},]
redpanda-1 | INFO 2026-03-12 15:05:41,036 [shard 0:main] raft - [group_id:2, {kafka/fulltext/0}] consensus.cc:1407 - Starting with voted_for {id: -2147483648, revision: -9223372036854775808} term 0 initial_state true
redpanda-1 | INFO 2026-03-12 15:05:41,047 [shard 0:main] raft - [group_id:3, {kafka/workspace/0}] consensus.cc:1407 - Starting with voted_for {id: -2147483648, revision: -9223372036854775808} term 0 initial_state true
redpanda-1 | INFO 2026-03-12 15:05:41,048 [shard 0:main] raft - [group_id:1, {kafka/tx/0}] consensus.cc:1407 - Starting with voted_for {id: -2147483648, revision: -9223372036854775808} term 0 initial_state true
redpanda-1 | INFO 2026-03-12 15:05:41,054 [shard 0:main] offset_translator - ntp: {kafka/fulltext/0} - offset_translator.cc:120 - resetting offset translation state
redpanda-1 | INFO 2026-03-12 15:05:41,064 [shard 0:main] offset_translator - ntp: {kafka/workspace/0} - offset_translator.cc:120 - resetting offset translation state
redpanda-1 | INFO 2026-03-12 15:05:41,066 [shard 0:main] offset_translator - ntp: {kafka/tx/0} - offset_translator.cc:120 - resetting offset translation state
redpanda-1 | INFO 2026-03-12 15:05:41,074 [shard 0:main] offset_translator - ntp: {kafka/fulltext/0} - offset_translator.cc:166 - started, state: {base offset/delta: -9223372036854775808/0, map size: 1, last delta: 0}, highest_known_offset: -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:41,076 [shard 0:main] raft - [group_id:2, {kafka/fulltext/0}] consensus.cc:1451 - Current log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, read bootstrap state: data_seen 0 config_seen 0 eol false commit 0 term 0 prev_idx 0 prev_term 0 config_tracker -9223372036854775808 commit_base_tracker -9223372036854775808 configurations []
redpanda-1 | INFO 2026-03-12 15:05:41,076 [shard 0:main] raft - [group_id:2, {kafka/fulltext/0}] consensus.cc:1478 - Truncating configurations at -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:41,084 [shard 0:main] offset_translator - ntp: {kafka/workspace/0} - offset_translator.cc:166 - started, state: {base offset/delta: -9223372036854775808/0, map size: 1, last delta: 0}, highest_known_offset: -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:41,084 [shard 0:main] offset_translator - ntp: {kafka/tx/0} - offset_translator.cc:166 - started, state: {base offset/delta: -9223372036854775808/0, map size: 1, last delta: 0}, highest_known_offset: -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:41,084 [shard 0:main] raft - [group_id:3, {kafka/workspace/0}] consensus.cc:1451 - Current log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, read bootstrap state: data_seen 0 config_seen 0 eol false commit 0 term 0 prev_idx 0 prev_term 0 config_tracker -9223372036854775808 commit_base_tracker -9223372036854775808 configurations []
redpanda-1 | INFO 2026-03-12 15:05:41,084 [shard 0:main] raft - [group_id:3, {kafka/workspace/0}] consensus.cc:1478 - Truncating configurations at -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:41,084 [shard 0:main] raft - [group_id:1, {kafka/tx/0}] consensus.cc:1451 - Current log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, read bootstrap state: data_seen 0 config_seen 0 eol false commit 0 term 0 prev_idx 0 prev_term 0 config_tracker -9223372036854775808 commit_base_tracker -9223372036854775808 configurations []
redpanda-1 | INFO 2026-03-12 15:05:41,084 [shard 0:main] raft - [group_id:1, {kafka/tx/0}] consensus.cc:1478 - Truncating configurations at -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:41,095 [shard 0:main] raft - [group_id:2, {kafka/fulltext/0}] consensus.cc:940 - starting pre-vote leader election, current term: 0, leadership transfer: false
redpanda-1 | INFO 2026-03-12 15:05:41,110 [shard 0:main] raft - [group_id:3, {kafka/workspace/0}] consensus.cc:940 - starting pre-vote leader election, current term: 0, leadership transfer: false
redpanda-1 | INFO 2026-03-12 15:05:41,110 [shard 0:main] raft - [group_id:1, {kafka/tx/0}] consensus.cc:940 - starting pre-vote leader election, current term: 0, leadership transfer: false
redpanda-1 | INFO 2026-03-12 15:05:41,121 [shard 0:main] raft - [group_id:2, {kafka/fulltext/0}] consensus.cc:1589 - started raft, log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, term: 0, configuration: {current: {voters: {{id: 0, revision: 7}}, learners: {}}, old:{nullopt}, revision: 7, update: {nullopt}, version: 5}}
redpanda-1 | INFO 2026-03-12 15:05:41,129 [shard 0:main] raft - [group_id:2, {kafka/fulltext/0}] vote_stm.cc:421 - becoming the leader term:1
redpanda-1 | INFO 2026-03-12 15:05:41,132 [shard 0:main] raft - [group_id:3, {kafka/workspace/0}] consensus.cc:1589 - started raft, log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, term: 0, configuration: {current: {voters: {{id: 0, revision: 9}}, learners: {}}, old:{nullopt}, revision: 9, update: {nullopt}, version: 5}}
redpanda-1 | INFO 2026-03-12 15:05:41,132 [shard 0:main] raft - [group_id:1, {kafka/tx/0}] consensus.cc:1589 - started raft, log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, term: 0, configuration: {current: {voters: {{id: 0, revision: 8}}, learners: {}}, old:{nullopt}, revision: 8, update: {nullopt}, version: 5}}
redpanda-1 | INFO 2026-03-12 15:05:41,132 [shard 0:main] storage - segment.cc:817 - Creating new segment /var/lib/redpanda/data/kafka/fulltext/0_7/0-1-v1.log
redpanda-1 | INFO 2026-03-12 15:05:41,142 [shard 0:main] raft - [group_id:3, {kafka/workspace/0}] vote_stm.cc:421 - becoming the leader term:1
redpanda-1 | INFO 2026-03-12 15:05:41,142 [shard 0:main] storage - segment.cc:817 - Creating new segment /var/lib/redpanda/data/kafka/workspace/0_9/0-1-v1.log
redpanda-1 | INFO 2026-03-12 15:05:41,142 [shard 0:main] raft - [group_id:1, {kafka/tx/0}] vote_stm.cc:421 - becoming the leader term:1
redpanda-1 | INFO 2026-03-12 15:05:41,142 [shard 0:main] storage - segment.cc:817 - Creating new segment /var/lib/redpanda/data/kafka/tx/0_8/0-1-v1.log
redpanda-1 | INFO 2026-03-12 15:05:41,153 [shard 0:main] tx - [{kafka/fulltext/0}] - rm_stm.cc:118 - Setting bootstrap committed offset to: 0
redpanda-1 | INFO 2026-03-12 15:05:41,153 [shard 0:main] raft - [group_id:2, {kafka/fulltext/0}] vote_stm.cc:436 - became the leader term: 1
redpanda-1 | INFO 2026-03-12 15:05:41,163 [shard 0:main] tx - [{kafka/workspace/0}] - rm_stm.cc:118 - Setting bootstrap committed offset to: 0
redpanda-1 | INFO 2026-03-12 15:05:41,163 [shard 0:main] tx - [{kafka/tx/0}] - rm_stm.cc:118 - Setting bootstrap committed offset to: 0
redpanda-1 | INFO 2026-03-12 15:05:41,163 [shard 0:main] raft - [group_id:3, {kafka/workspace/0}] vote_stm.cc:436 - became the leader term: 1
redpanda-1 | INFO 2026-03-12 15:05:41,163 [shard 0:main] raft - [group_id:1, {kafka/tx/0}] vote_stm.cc:436 - became the leader term: 1
redpanda-1 | INFO 2026-03-12 15:05:41,244 [shard 0:main] cluster - topics_frontend.cc:346 - Create topics [{configuration: { topic: {kafka/__consumer_offsets}, partition_count: 3, replication_factor: 1, is_migrated: false, properties: { compression: {nullopt}, cleanup_policy_bitflags: {compact}, compaction_strategy: {nullopt}, retention_bytes: {{nullopt}}, retention_duration_ms: {{nullopt}}, segment_size: {nullopt}, timestamp_type: {nullopt}, recovery_enabled: {nullopt}, shadow_indexing: {disabled}, read_replica: {nullopt}, read_replica_bucket: {nullopt}, remote_topic_namespace_override: {nullopt}, remote_topic_properties: {nullopt}, batch_max_bytes: {nullopt}, retention_local_target_bytes: {{nullopt}}, retention_local_target_ms: {{nullopt}}, remote_delete: true, segment_ms: {{nullopt}}, record_key_schema_id_validation: {nullopt}, record_key_schema_id_validation_compat: {nullopt}, record_key_subject_name_strategy: {nullopt}, record_key_subject_name_strategy_compat: {nullopt}, record_value_schema_id_validation: {nullopt}, record_value_schema_id_validation_compat: {nullopt}, record_value_subject_name_strategy: {nullopt}, record_value_subject_name_strategy_compat: {nullopt}, initial_retention_local_target_bytes: {{nullopt}}, initial_retention_local_target_ms: {{nullopt}}, mpx_virtual_cluster_id: {nullopt}, write_caching: {nullopt}, flush_ms: {nullopt}, flush_bytes: {nullopt}, remote_label: {nullopt}, iceberg_mode: disabled, leaders_preference: {nullopt}, delete_retention_ms: {disabled}, iceberg_delete: {nullopt}}}, custom_assignments: {}},]
redpanda-1 | INFO 2026-03-12 15:05:41,245 [shard 0:main] cluster - topics_frontend.cc:346 - Create topics [{configuration: { topic: {kafka/__consumer_offsets}, partition_count: 3, replication_factor: 1, is_migrated: false, properties: { compression: {nullopt}, cleanup_policy_bitflags: {compact}, compaction_strategy: {nullopt}, retention_bytes: {{nullopt}}, retention_duration_ms: {{nullopt}}, segment_size: {nullopt}, timestamp_type: {nullopt}, recovery_enabled: {nullopt}, shadow_indexing: {disabled}, read_replica: {nullopt}, read_replica_bucket: {nullopt}, remote_topic_namespace_override: {nullopt}, remote_topic_properties: {nullopt}, batch_max_bytes: {nullopt}, retention_local_target_bytes: {{nullopt}}, retention_local_target_ms: {{nullopt}}, remote_delete: true, segment_ms: {{nullopt}}, record_key_schema_id_validation: {nullopt}, record_key_schema_id_validation_compat: {nullopt}, record_key_subject_name_strategy: {nullopt}, record_key_subject_name_strategy_compat: {nullopt}, record_value_schema_id_validation: {nullopt}, record_value_schema_id_validation_compat: {nullopt}, record_value_subject_name_strategy: {nullopt}, record_value_subject_name_strategy_compat: {nullopt}, initial_retention_local_target_bytes: {{nullopt}}, initial_retention_local_target_ms: {{nullopt}}, mpx_virtual_cluster_id: {nullopt}, write_caching: {nullopt}, flush_ms: {nullopt}, flush_bytes: {nullopt}, remote_label: {nullopt}, iceberg_mode: disabled, leaders_preference: {nullopt}, delete_retention_ms: {disabled}, iceberg_delete: {nullopt}}}, custom_assignments: {}},]
redpanda-1 | INFO 2026-03-12 15:05:41,245 [shard 0:main] cluster - topics_frontend.cc:346 - Create topics [{configuration: { topic: {kafka/__consumer_offsets}, partition_count: 3, replication_factor: 1, is_migrated: false, properties: { compression: {nullopt}, cleanup_policy_bitflags: {compact}, compaction_strategy: {nullopt}, retention_bytes: {{nullopt}}, retention_duration_ms: {{nullopt}}, segment_size: {nullopt}, timestamp_type: {nullopt}, recovery_enabled: {nullopt}, shadow_indexing: {disabled}, read_replica: {nullopt}, read_replica_bucket: {nullopt}, remote_topic_namespace_override: {nullopt}, remote_topic_properties: {nullopt}, batch_max_bytes: {nullopt}, retention_local_target_bytes: {{nullopt}}, retention_local_target_ms: {{nullopt}}, remote_delete: true, segment_ms: {{nullopt}}, record_key_schema_id_validation: {nullopt}, record_key_schema_id_validation_compat: {nullopt}, record_key_subject_name_strategy: {nullopt}, record_key_subject_name_strategy_compat: {nullopt}, record_value_schema_id_validation: {nullopt}, record_value_schema_id_validation_compat: {nullopt}, record_value_subject_name_strategy: {nullopt}, record_value_subject_name_strategy_compat: {nullopt}, initial_retention_local_target_bytes: {{nullopt}}, initial_retention_local_target_ms: {{nullopt}}, mpx_virtual_cluster_id: {nullopt}, write_caching: {nullopt}, flush_ms: {nullopt}, flush_bytes: {nullopt}, remote_label: {nullopt}, iceberg_mode: disabled, leaders_preference: {nullopt}, delete_retention_ms: {disabled}, iceberg_delete: {nullopt}}}, custom_assignments: {}},]
redpanda-1 | INFO 2026-03-12 15:05:41,296 [shard 0:main] raft - [group_id:4, {kafka/__consumer_offsets/0}] consensus.cc:1407 - Starting with voted_for {id: -2147483648, revision: -9223372036854775808} term 0 initial_state true
redpanda-1 | INFO 2026-03-12 15:05:41,296 [shard 0:main] raft - [group_id:5, {kafka/__consumer_offsets/1}] consensus.cc:1407 - Starting with voted_for {id: -2147483648, revision: -9223372036854775808} term 0 initial_state true
redpanda-1 | INFO 2026-03-12 15:05:41,297 [shard 0:main] raft - [group_id:6, {kafka/__consumer_offsets/2}] consensus.cc:1407 - Starting with voted_for {id: -2147483648, revision: -9223372036854775808} term 0 initial_state true
redpanda-1 | INFO 2026-03-12 15:05:41,316 [shard 0:main] offset_translator - ntp: {kafka/__consumer_offsets/0} - offset_translator.cc:120 - resetting offset translation state
redpanda-1 | INFO 2026-03-12 15:05:41,316 [shard 0:main] offset_translator - ntp: {kafka/__consumer_offsets/1} - offset_translator.cc:120 - resetting offset translation state
redpanda-1 | INFO 2026-03-12 15:05:41,316 [shard 0:main] offset_translator - ntp: {kafka/__consumer_offsets/2} - offset_translator.cc:120 - resetting offset translation state
redpanda-1 | INFO 2026-03-12 15:05:41,336 [shard 0:main] offset_translator - ntp: {kafka/__consumer_offsets/0} - offset_translator.cc:166 - started, state: {base offset/delta: -9223372036854775808/0, map size: 1, last delta: 0}, highest_known_offset: -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:41,337 [shard 0:main] offset_translator - ntp: {kafka/__consumer_offsets/1} - offset_translator.cc:166 - started, state: {base offset/delta: -9223372036854775808/0, map size: 1, last delta: 0}, highest_known_offset: -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:41,337 [shard 0:main] offset_translator - ntp: {kafka/__consumer_offsets/2} - offset_translator.cc:166 - started, state: {base offset/delta: -9223372036854775808/0, map size: 1, last delta: 0}, highest_known_offset: -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:41,337 [shard 0:main] raft - [group_id:4, {kafka/__consumer_offsets/0}] consensus.cc:1451 - Current log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, read bootstrap state: data_seen 0 config_seen 0 eol false commit 0 term 0 prev_idx 0 prev_term 0 config_tracker -9223372036854775808 commit_base_tracker -9223372036854775808 configurations []
redpanda-1 | INFO 2026-03-12 15:05:41,337 [shard 0:main] raft - [group_id:4, {kafka/__consumer_offsets/0}] consensus.cc:1478 - Truncating configurations at -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:41,337 [shard 0:main] raft - [group_id:5, {kafka/__consumer_offsets/1}] consensus.cc:1451 - Current log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, read bootstrap state: data_seen 0 config_seen 0 eol false commit 0 term 0 prev_idx 0 prev_term 0 config_tracker -9223372036854775808 commit_base_tracker -9223372036854775808 configurations []
redpanda-1 | INFO 2026-03-12 15:05:41,337 [shard 0:main] raft - [group_id:5, {kafka/__consumer_offsets/1}] consensus.cc:1478 - Truncating configurations at -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:41,337 [shard 0:main] raft - [group_id:6, {kafka/__consumer_offsets/2}] consensus.cc:1451 - Current log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, read bootstrap state: data_seen 0 config_seen 0 eol false commit 0 term 0 prev_idx 0 prev_term 0 config_tracker -9223372036854775808 commit_base_tracker -9223372036854775808 configurations []
redpanda-1 | INFO 2026-03-12 15:05:41,337 [shard 0:main] raft - [group_id:6, {kafka/__consumer_offsets/2}] consensus.cc:1478 - Truncating configurations at -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:05:41,357 [shard 0:main] raft - [group_id:4, {kafka/__consumer_offsets/0}] consensus.cc:940 - starting pre-vote leader election, current term: 0, leadership transfer: false
redpanda-1 | INFO 2026-03-12 15:05:41,357 [shard 0:main] raft - [group_id:5, {kafka/__consumer_offsets/1}] consensus.cc:940 - starting pre-vote leader election, current term: 0, leadership transfer: false
redpanda-1 | INFO 2026-03-12 15:05:41,357 [shard 0:main] raft - [group_id:6, {kafka/__consumer_offsets/2}] consensus.cc:940 - starting pre-vote leader election, current term: 0, leadership transfer: false
redpanda-1 | INFO 2026-03-12 15:05:41,367 [shard 0:main] raft - [group_id:4, {kafka/__consumer_offsets/0}] consensus.cc:1589 - started raft, log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, term: 0, configuration: {current: {voters: {{id: 0, revision: 10}}, learners: {}}, old:{nullopt}, revision: 10, update: {nullopt}, version: 5}}
redpanda-1 | INFO 2026-03-12 15:05:41,367 [shard 0:main] raft - [group_id:5, {kafka/__consumer_offsets/1}] consensus.cc:1589 - started raft, log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, term: 0, configuration: {current: {voters: {{id: 0, revision: 10}}, learners: {}}, old:{nullopt}, revision: 10, update: {nullopt}, version: 5}}
redpanda-1 | INFO 2026-03-12 15:05:41,369 [shard 0:main] raft - [group_id:6, {kafka/__consumer_offsets/2}] consensus.cc:1589 - started raft, log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, term: 0, configuration: {current: {voters: {{id: 0, revision: 10}}, learners: {}}, old:{nullopt}, revision: 10, update: {nullopt}, version: 5}}
redpanda-1 | INFO 2026-03-12 15:05:41,379 [shard 0:main] raft - [group_id:4, {kafka/__consumer_offsets/0}] vote_stm.cc:421 - becoming the leader term:1
redpanda-1 | INFO 2026-03-12 15:05:41,379 [shard 0:main] storage - segment.cc:817 - Creating new segment /var/lib/redpanda/data/kafka/__consumer_offsets/0_10/0-1-v1.log
redpanda-1 | INFO 2026-03-12 15:05:41,379 [shard 0:main] raft - [group_id:5, {kafka/__consumer_offsets/1}] vote_stm.cc:421 - becoming the leader term:1
redpanda-1 | INFO 2026-03-12 15:05:41,379 [shard 0:main] storage - segment.cc:817 - Creating new segment /var/lib/redpanda/data/kafka/__consumer_offsets/1_10/0-1-v1.log
redpanda-1 | INFO 2026-03-12 15:05:41,379 [shard 0:main] raft - [group_id:6, {kafka/__consumer_offsets/2}] vote_stm.cc:421 - becoming the leader term:1
redpanda-1 | INFO 2026-03-12 15:05:41,379 [shard 0:main] storage - segment.cc:817 - Creating new segment /var/lib/redpanda/data/kafka/__consumer_offsets/2_10/0-1-v1.log
redpanda-1 | INFO 2026-03-12 15:05:41,409 [shard 0:main] raft - [group_id:4, {kafka/__consumer_offsets/0}] vote_stm.cc:436 - became the leader term: 1
redpanda-1 | INFO 2026-03-12 15:05:41,409 [shard 0:main] raft - [group_id:5, {kafka/__consumer_offsets/1}] vote_stm.cc:436 - became the leader term: 1
redpanda-1 | INFO 2026-03-12 15:05:41,409 [shard 0:main] raft - [group_id:6, {kafka/__consumer_offsets/2}] vote_stm.cc:436 - became the leader term: 1
redpanda-1 | INFO 2026-03-12 15:05:41,419 [shard 0:main] kafka - group_manager.cc:912 - Scheduling write of offset retention feature fence for partition {log:{offsets: {start_offset:0, committed_offset:0, committed_offset_term:1, dirty_offset:0, dirty_offset_term:1}, is_closed: false, segments: [{size: 1, [{offset_tracker:{term:1, base_offset:0, committed_offset:0, dirty_offset:0}, compacted_segment=1, finished_self_compaction=0, finished_windowed_compaction=0, generation=2, reader={/var/lib/redpanda/data/kafka/__consumer_offsets/0_10/0-1-v1.log, (116 bytes)}, writer={no_of_chunks:64, closed:0, fallocation_offset:33554432, stable_offset:116, flushed_offset:116, committed_offset:116, inflight:0, dispatched:0, merged:0, bytes_flush_pending:0}, cache=nullptr, compaction_index:{name:/var/lib/redpanda/data/kafka/__consumer_offsets/0_10/0-1-v1.compaction_index, key_mem_usage:0, persisted_entries:0, in_memory_entries:0, file_appender:{nullopt}}, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/__consumer_offsets/0_10/0-1-v1.base_index, offsets:0, index:{header_bitflags:0, base_offset:0, max_offset:0, base_timestamp:{timestamp: 1773327941379}, max_timestamp:{timestamp: 1773327941379}, batch_timestamps_are_monotonic:1, with_offset:true, non_data_timestamps:1, broker_timestamp:{nullopt}, num_compactible_records_appended:{0}, clean_compact_timestamp:{nullopt}, may_have_tombstone_records:1, index(1, 1, 1)}, step:32768, needs_persistence:1}}]}], config: {ntp: {kafka/__consumer_offsets/0}, base_dir: /var/lib/redpanda/data, overrides: {compaction_strategy: {nullopt}, cleanup_policy_bitflags: {compact}, segment_size: {nullopt}, retention_bytes: {{nullopt}}, retention_time_ms: {{nullopt}}, recovery_enabled: false, retention_local_target_bytes: {{nullopt}}, retention_local_target_ms: {{nullopt}}, remote_delete: {true}, segment_ms: {{nullopt}}, initial_retention_local_target_bytes: {{nullopt}}, initial_retention_local_target_ms: {{nullopt}}, write_caching: {nullopt}, flush_ms: {nullopt}, flush_bytes: {nullopt} iceberg_mode: disabled }}, revision: 10, topic_revision: 10, remote_revision: 10}}, group_id:4, term: 1, commit_index: 0, voted_for:{id: 0, revision: 10}}
redpanda-1 | INFO 2026-03-12 15:05:41,422 [shard 0:main] kafka - group_manager.cc:912 - Scheduling write of offset retention feature fence for partition {log:{offsets: {start_offset:0, committed_offset:0, committed_offset_term:1, dirty_offset:0, dirty_offset_term:1}, is_closed: false, segments: [{size: 1, [{offset_tracker:{term:1, base_offset:0, committed_offset:0, dirty_offset:0}, compacted_segment=1, finished_self_compaction=0, finished_windowed_compaction=0, generation=2, reader={/var/lib/redpanda/data/kafka/__consumer_offsets/1_10/0-1-v1.log, (116 bytes)}, writer={no_of_chunks:64, closed:0, fallocation_offset:33554432, stable_offset:116, flushed_offset:116, committed_offset:116, inflight:0, dispatched:0, merged:0, bytes_flush_pending:0}, cache=nullptr, compaction_index:{name:/var/lib/redpanda/data/kafka/__consumer_offsets/1_10/0-1-v1.compaction_index, key_mem_usage:0, persisted_entries:0, in_memory_entries:0, file_appender:{nullopt}}, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/__consumer_offsets/1_10/0-1-v1.base_index, offsets:0, index:{header_bitflags:0, base_offset:0, max_offset:0, base_timestamp:{timestamp: 1773327941379}, max_timestamp:{timestamp: 1773327941379}, batch_timestamps_are_monotonic:1, with_offset:true, non_data_timestamps:1, broker_timestamp:{nullopt}, num_compactible_records_appended:{0}, clean_compact_timestamp:{nullopt}, may_have_tombstone_records:1, index(1, 1, 1)}, step:32768, needs_persistence:1}}]}], config: {ntp: {kafka/__consumer_offsets/1}, base_dir: /var/lib/redpanda/data, overrides: {compaction_strategy: {nullopt}, cleanup_policy_bitflags: {compact}, segment_size: {nullopt}, retention_bytes: {{nullopt}}, retention_time_ms: {{nullopt}}, recovery_enabled: false, retention_local_target_bytes: {{nullopt}}, retention_local_target_ms: {{nullopt}}, remote_delete: {true}, segment_ms: {{nullopt}}, initial_retention_local_target_bytes: {{nullopt}}, initial_retention_local_target_ms: {{nullopt}}, write_caching: {nullopt}, flush_ms: {nullopt}, flush_bytes: {nullopt} iceberg_mode: disabled }}, revision: 10, topic_revision: 10, remote_revision: 10}}, group_id:5, term: 1, commit_index: 0, voted_for:{id: 0, revision: 10}}
redpanda-1 | INFO 2026-03-12 15:05:41,422 [shard 0:main] kafka - group_manager.cc:912 - Scheduling write of offset retention feature fence for partition {log:{offsets: {start_offset:0, committed_offset:0, committed_offset_term:1, dirty_offset:0, dirty_offset_term:1}, is_closed: false, segments: [{size: 1, [{offset_tracker:{term:1, base_offset:0, committed_offset:0, dirty_offset:0}, compacted_segment=1, finished_self_compaction=0, finished_windowed_compaction=0, generation=2, reader={/var/lib/redpanda/data/kafka/__consumer_offsets/2_10/0-1-v1.log, (116 bytes)}, writer={no_of_chunks:64, closed:0, fallocation_offset:33554432, stable_offset:116, flushed_offset:116, committed_offset:116, inflight:0, dispatched:0, merged:0, bytes_flush_pending:0}, cache=nullptr, compaction_index:{name:/var/lib/redpanda/data/kafka/__consumer_offsets/2_10/0-1-v1.compaction_index, key_mem_usage:0, persisted_entries:0, in_memory_entries:0, file_appender:{nullopt}}, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/__consumer_offsets/2_10/0-1-v1.base_index, offsets:0, index:{header_bitflags:0, base_offset:0, max_offset:0, base_timestamp:{timestamp: 1773327941379}, max_timestamp:{timestamp: 1773327941379}, batch_timestamps_are_monotonic:1, with_offset:true, non_data_timestamps:1, broker_timestamp:{nullopt}, num_compactible_records_appended:{0}, clean_compact_timestamp:{nullopt}, may_have_tombstone_records:1, index(1, 1, 1)}, step:32768, needs_persistence:1}}]}], config: {ntp: {kafka/__consumer_offsets/2}, base_dir: /var/lib/redpanda/data, overrides: {compaction_strategy: {nullopt}, cleanup_policy_bitflags: {compact}, segment_size: {nullopt}, retention_bytes: {{nullopt}}, retention_time_ms: {{nullopt}}, recovery_enabled: false, retention_local_target_bytes: {{nullopt}}, retention_local_target_ms: {{nullopt}}, remote_delete: {true}, segment_ms: {{nullopt}}, initial_retention_local_target_bytes: {{nullopt}}, initial_retention_local_target_ms: {{nullopt}}, write_caching: {nullopt}, flush_ms: {nullopt}, flush_bytes: {nullopt} iceberg_mode: disabled }}, revision: 10, topic_revision: 10, remote_revision: 10}}, group_id:6, term: 1, commit_index: 0, voted_for:{id: 0, revision: 10}}
redpanda-1 | INFO 2026-03-12 15:05:41,422 [shard 0:main] kafka - group_manager.cc:1041 - Prepared partition {kafka/__consumer_offsets/0} for consumer offset retention feature during upgrade
redpanda-1 | INFO 2026-03-12 15:05:41,422 [shard 0:main] kafka - group_manager.cc:1041 - Prepared partition {kafka/__consumer_offsets/1} for consumer offset retention feature during upgrade
redpanda-1 | INFO 2026-03-12 15:05:41,422 [shard 0:main] kafka - group_manager.cc:1041 - Prepared partition {kafka/__consumer_offsets/2} for consumer offset retention feature during upgrade
redpanda-1 | INFO 2026-03-12 15:06:18,354 [shard 0:main] cluster - controller_stm.cc:138 - created snapshot at offset 12 in 2 ms
redpanda-1 | INFO 2026-03-12 15:06:18,388 [shard 0:main] storage - disk_log_impl.cc:2555 - Removing "/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log" (remove_prefix_full_segments, {offset_tracker:{term:1, base_offset:0, committed_offset:12, dirty_offset:12}, compacted_segment=0, finished_self_compaction=0, finished_windowed_compaction=0, generation=25, reader={/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log, (2966 bytes)}, writer={no_of_chunks:64, closed:0, fallocation_offset:33554432, stable_offset:2966, flushed_offset:2966, committed_offset:2966, inflight:0, dispatched:0, merged:1, bytes_flush_pending:0}, cache={cache_size=13, dirty tracker: {min: -9223372036854775808, max: -9223372036854775808}}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.base_index, offsets:0, index:{header_bitflags:0, base_offset:0, max_offset:12, base_timestamp:{timestamp: 1773327918275}, max_timestamp:{timestamp: 1773327941245}, batch_timestamps_are_monotonic:1, with_offset:true, non_data_timestamps:0, broker_timestamp:{{timestamp: 1773327941245}}, num_compactible_records_appended:{12}, clean_compact_timestamp:{nullopt}, may_have_tombstone_records:1, index(1, 1, 1)}, step:32768, needs_persistence:1}})
redpanda-1 | INFO 2026-03-12 15:06:18,389 [shard 0:main] storage - disk_log_impl.cc:2564 - Segment has outstanding locks. Might take a while to close:/var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log
redpanda-1 | INFO 2026-03-12 15:06:30,050 [shard 0:main] cluster - topics_frontend.cc:346 - Create topics [{configuration: { topic: {kafka/process}, partition_count: 1, replication_factor: 1, is_migrated: false, properties: { compression: {nullopt}, cleanup_policy_bitflags: {nullopt}, compaction_strategy: {nullopt}, retention_bytes: {{nullopt}}, retention_duration_ms: {{nullopt}}, segment_size: {nullopt}, timestamp_type: {nullopt}, recovery_enabled: {nullopt}, shadow_indexing: {disabled}, read_replica: {nullopt}, read_replica_bucket: {nullopt}, remote_topic_namespace_override: {nullopt}, remote_topic_properties: {nullopt}, batch_max_bytes: {nullopt}, retention_local_target_bytes: {{nullopt}}, retention_local_target_ms: {{nullopt}}, remote_delete: true, segment_ms: {{nullopt}}, record_key_schema_id_validation: {nullopt}, record_key_schema_id_validation_compat: {nullopt}, record_key_subject_name_strategy: {nullopt}, record_key_subject_name_strategy_compat: {nullopt}, record_value_schema_id_validation: {nullopt}, record_value_schema_id_validation_compat: {nullopt}, record_value_subject_name_strategy: {nullopt}, record_value_subject_name_strategy_compat: {nullopt}, initial_retention_local_target_bytes: {{nullopt}}, initial_retention_local_target_ms: {{nullopt}}, mpx_virtual_cluster_id: {nullopt}, write_caching: {nullopt}, flush_ms: {nullopt}, flush_bytes: {nullopt}, remote_label: {nullopt}, iceberg_mode: disabled, leaders_preference: {nullopt}, delete_retention_ms: {disabled}, iceberg_delete: {nullopt}}}, custom_assignments: {}},]
redpanda-1 | INFO 2026-03-12 15:06:30,050 [shard 0:main] storage - segment.cc:817 - Creating new segment /var/lib/redpanda/data/redpanda/controller/0_0/13-1-v1.log
redpanda-1 | INFO 2026-03-12 15:06:30,093 [shard 0:main] raft - [group_id:13, {kafka/process/0}] consensus.cc:1407 - Starting with voted_for {id: -2147483648, revision: -9223372036854775808} term 0 initial_state true
redpanda-1 | INFO 2026-03-12 15:06:30,113 [shard 0:main] offset_translator - ntp: {kafka/process/0} - offset_translator.cc:120 - resetting offset translation state
redpanda-1 | INFO 2026-03-12 15:06:30,133 [shard 0:main] offset_translator - ntp: {kafka/process/0} - offset_translator.cc:166 - started, state: {base offset/delta: -9223372036854775808/0, map size: 1, last delta: 0}, highest_known_offset: -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:06:30,133 [shard 0:main] raft - [group_id:13, {kafka/process/0}] consensus.cc:1451 - Current log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, read bootstrap state: data_seen 0 config_seen 0 eol false commit 0 term 0 prev_idx 0 prev_term 0 config_tracker -9223372036854775808 commit_base_tracker -9223372036854775808 configurations []
redpanda-1 | INFO 2026-03-12 15:06:30,133 [shard 0:main] raft - [group_id:13, {kafka/process/0}] consensus.cc:1478 - Truncating configurations at -9223372036854775808
redpanda-1 | INFO 2026-03-12 15:06:30,154 [shard 0:main] raft - [group_id:13, {kafka/process/0}] consensus.cc:940 - starting pre-vote leader election, current term: 0, leadership transfer: false
redpanda-1 | INFO 2026-03-12 15:06:30,174 [shard 0:main] raft - [group_id:13, {kafka/process/0}] consensus.cc:1589 - started raft, log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808}, term: 0, configuration: {current: {voters: {{id: 0, revision: 13}}, learners: {}}, old:{nullopt}, revision: 13, update: {nullopt}, version: 5}}
redpanda-1 | INFO 2026-03-12 15:06:30,185 [shard 0:main] raft - [group_id:13, {kafka/process/0}] vote_stm.cc:421 - becoming the leader term:1
redpanda-1 | INFO 2026-03-12 15:06:30,185 [shard 0:main] storage - segment.cc:817 - Creating new segment /var/lib/redpanda/data/kafka/process/0_13/0-1-v1.log
redpanda-1 | INFO 2026-03-12 15:06:30,206 [shard 0:main] tx - [{kafka/process/0}] - rm_stm.cc:118 - Setting bootstrap committed offset to: 0
redpanda-1 | INFO 2026-03-12 15:06:30,206 [shard 0:main] raft - [group_id:13, {kafka/process/0}] vote_stm.cc:436 - became the leader term: 1
redpanda-1 | INFO 2026-03-12 15:07:30,051 [shard 0:main] cluster - controller_stm.cc:138 - created snapshot at offset 13 in 0 ms
redpanda-1 | INFO 2026-03-12 15:07:30,062 [shard 0:main] storage - disk_log_impl.cc:2555 - Removing "/var/lib/redpanda/data/redpanda/controller/0_0/13-1-v1.log" (remove_prefix_full_segments, {offset_tracker:{term:1, base_offset:13, committed_offset:13, dirty_offset:13}, compacted_segment=0, finished_self_compaction=0, finished_windowed_compaction=0, generation=2, reader={/var/lib/redpanda/data/redpanda/controller/0_0/13-1-v1.log, (238 bytes)}, writer={no_of_chunks:64, closed:0, fallocation_offset:33554432, stable_offset:238, flushed_offset:238, committed_offset:238, inflight:0, dispatched:0, merged:0, bytes_flush_pending:0}, cache={cache_size=1, dirty tracker: {min: -9223372036854775808, max: -9223372036854775808}}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/redpanda/controller/0_0/13-1-v1.base_index, offsets:13, index:{header_bitflags:0, base_offset:13, max_offset:13, base_timestamp:{timestamp: 1773327990050}, max_timestamp:{timestamp: 1773327990050}, batch_timestamps_are_monotonic:1, with_offset:true, non_data_timestamps:0, broker_timestamp:{{timestamp: 1773327990051}}, num_compactible_records_appended:{1}, clean_compact_timestamp:{nullopt}, may_have_tombstone_records:1, index(1, 1, 1)}, step:32768, needs_persistence:1}})
redpanda-1 | INFO 2026-03-12 15:07:30,062 [shard 0:main] storage - disk_log_impl.cc:2564 - Segment has outstanding locks. Might take a while to close:/var/lib/redpanda/data/redpanda/controller/0_0/13-1-v1.log
```

Docker compose:

```yml
name: ${DOCKER_NAME}
services:
nginx:
image: "nginx:1.21.3"
ports:
- "${HTTP_BIND}:${HTTP_PORT}:80"
volumes:
- ./.huly.nginx:/etc/nginx/conf.d/default.conf
restart: unless-stopped
networks:
- huly_net

cockroach:
image: cockroachdb/cockroach:latest-v24.2
command: start-single-node --accept-sql-without-tls
environment:
- COCKROACH_DATABASE=${CR_DATABASE}
- COCKROACH_USER=${CR_USERNAME}
- COCKROACH_PASSWORD=${CR_USER_PASSWORD}
volumes:
- ${VOLUME_CR_DATA_PATH:-cr_data}:/cockroach/cockroach-data
- ${VOLUME_CR_CERTS_PATH:-cr_certs}:/cockroach/certs
restart: unless-stopped
networks:
- huly_net

redpanda:
image: docker.redpanda.com/redpandadata/redpanda:v24.3.6
command:
- redpanda
- start
- --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
- --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092
- --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
- --advertise-pandaproxy-addr internal://redpanda:8082,external://localhost:18082
- --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
- --rpc-addr redpanda:33145
- --advertise-rpc-addr redpanda:33145
- --mode dev-container
- --smp 1
- --default-log-level=info
volumes:
- ${VOLUME_REDPANDA_PATH:-redpanda}:/var/lib/redpanda/data
environment:
- REDPANDA_SUPERUSER_USERNAME=${REDPANDA_ADMIN_USER}
- REDPANDA_SUPERUSER_PASSWORD=${REDPANDA_ADMIN_PWD}
healthcheck:
test: ['CMD', 'rpk', 'cluster', 'info', '-X', 'user=${REDPANDA_ADMIN_USER}', '-X', 'pass=${REDPANDA_ADMIN_PWD}']
interval: 10s
timeout: 5s
retries: 10
networks:
- huly_net

minio:
image: "minio/minio"
command: server /data --address ":9000" --console-address ":9001"
volumes:
- ${VOLUME_FILES_PATH:-files}:/data
healthcheck:
test: ['CMD', 'mc', 'ready', 'local']
interval: 5s
retries: 10
restart: unless-stopped
networks:
- huly_net

elastic:
image: "elasticsearch:7.14.2"
command: |
/bin/sh -c "./bin/elasticsearch-plugin list | grep -q ingest-attachment || yes | ./bin/elasticsearch-plugin install --silent ingest-attachment;
/usr/local/bin/docker-entrypoint.sh eswrapper"
volumes:
- ${VOLUME_ELASTIC_PATH:-elastic}:/usr/share/elasticsearch/data
environment:
- ELASTICSEARCH_PORT_NUMBER=9200
- BITNAMI_DEBUG=true
- discovery.type=single-node
- ES_JAVA_OPTS=-Xms1024m -Xmx1024m
- http.cors.enabled=true
- http.cors.allow-origin=http://localhost:8082
healthcheck:
interval: 20s
retries: 10
test: curl -s http://localhost:9200/_cluster/health | grep -vq '"status":"red"'
restart: unless-stopped
networks:
- huly_net

# mongodb:
# image: mongo:7-jammy
# restart: unless-stopped
# volumes:
# - ${VOLUME_MONGO_PATH:-mongodb}:/data/db
# networks:
# - huly_net
# # Required by aibot and calendar services

# redis:
# image: redis:7-alpine
# command: redis-server --maxmemory 512mb --maxmemory-policy allkeys-lru
# restart: unless-stopped
# networks:
# - huly_net
# # Required by HulyPulse (push notifications / real-time updates)

# hulypulse:
# image: hardcoreeng/service_hulypulse:${HULY_PULSE_VERSION:-0.1.29}
# ports:
# - 8099:8099
# environment:
# - HULY_BIND_PORT=8099
# - HULY_LOG=info
# - HULY_TOKEN_SECRET=${SECRET}
# - HULY_HEARTBEAT_TIMEOUT=60
# - HULY_BACKEND=memory
# # For Redis backend instead of in-memory (optional, e.g. multi-node setups):
# # - HULY_BACKEND=redis
# # - HULY_REDIS_MODE=direct
# # - HULY_REDIS_URLS=redis://redis:6379
# deploy:
# resources:
# limits:
# memory: 512M
# restart: unless-stopped
# networks:
# - huly_net

rekoni:
image: hardcoreeng/rekoni-service:${HULY_VERSION}
environment:
- SECRET=${SECRET}
deploy:
resources:
limits:
memory: 500M
restart: unless-stopped
networks:
- huly_net

transactor:
image: hardcoreeng/transactor:${HULY_VERSION}
environment:
- SERVER_PORT=3333
- SERVER_SECRET=${SECRET}
- DB_URL=${CR_DB_URL}
- STORAGE_CONFIG=minio|minio?accessKey=minioadmin&secretKey=minioadmin
- FRONT_URL=http://localhost:8087
- ACCOUNTS_URL=http://account:3000
- FULLTEXT_URL=http://fulltext:4700
- STATS_URL=http://stats:4900
- LAST_NAME_FIRST=${LAST_NAME_FIRST:-true}
- QUEUE_CONFIG=redpanda:9092
restart: unless-stopped
networks:
- huly_net

collaborator:
image: hardcoreeng/collaborator:${HULY_VERSION}
environment:
- COLLABORATOR_PORT=3078
- SECRET=${SECRET}
- ACCOUNTS_URL=http://account:3000
- STATS_URL=http://stats:4900
- STORAGE_CONFIG=minio|minio?accessKey=minioadmin&secretKey=minioadmin
restart: unless-stopped
networks:
- huly_net

account:
image: hardcoreeng/account:${HULY_VERSION}
environment:
- SERVER_PORT=3000
- SERVER_SECRET=${SECRET}
- DB_URL=${CR_DB_URL}
- TRANSACTOR_URL=ws://transactor:3333;ws${SECURE:+s}://${HOST_ADDRESS}/_transactor
- STORAGE_CONFIG=minio|minio?accessKey=minioadmin&secretKey=minioadmin
- FRONT_URL=http${SECURE:+s}://${HOST_ADDRESS}
- STATS_URL=http${SECURE:+s}://${HOST_ADDRESS}/_stats
- MODEL_ENABLED=*
- ACCOUNTS_URL=http${SECURE:+s}://${HOST_ADDRESS}/_accounts
- ACCOUNT_PORT=3000
- QUEUE_CONFIG=redpanda:9092
restart: unless-stopped
networks:
- huly_net

workspace:
image: hardcoreeng/workspace:${HULY_VERSION}
environment:
- SERVER_SECRET=${SECRET}
- DB_URL=${CR_DB_URL}
- TRANSACTOR_URL=ws://transactor:3333;ws${SECURE:+s}://${HOST_ADDRESS}/_transactor
- STORAGE_CONFIG=minio|minio?accessKey=minioadmin&secretKey=minioadmin
- MODEL_ENABLED=*
- ACCOUNTS_URL=http://account:3000
- STATS_URL=http://stats:4900
- QUEUE_CONFIG=redpanda:9092
- ACCOUNTS_DB_URL=${CR_DB_URL}
restart: unless-stopped
networks:
- huly_net

front:
image: hardcoreeng/front:${HULY_VERSION}
environment:
- SERVER_PORT=8080
- SERVER_SECRET=${SECRET}
- LOVE_ENDPOINT=http${SECURE:+s}://${HOST_ADDRESS}/_love
- ACCOUNTS_URL=http${SECURE:+s}://${HOST_ADDRESS}/_accounts
- ACCOUNTS_URL_INTERNAL=http://account:3000
- REKONI_URL=http${SECURE:+s}://${HOST_ADDRESS}/_rekoni
- CALENDAR_URL=http${SECURE:+s}://${HOST_ADDRESS}/_calendar
- GMAIL_URL=http${SECURE:+s}://${HOST_ADDRESS}/_gmail
- TELEGRAM_URL=http${SECURE:+s}://${HOST_ADDRESS}/_telegram
- STATS_URL=http${SECURE:+s}://${HOST_ADDRESS}/_stats
- UPLOAD_URL=/files
- ELASTIC_URL=http://elastic:9200
- COLLABORATOR_URL=ws${SECURE:+s}://${HOST_ADDRESS}/_collaborator
- STORAGE_CONFIG=minio|minio?accessKey=minioadmin&secretKey=minioadmin
- TITLE=${TITLE:-Huly Self Host}
- DEFAULT_LANGUAGE=${DEFAULT_LANGUAGE:-en}
- LAST_NAME_FIRST=${LAST_NAME_FIRST:-true}
- DESKTOP_UPDATES_CHANNEL=${DESKTOP_CHANNEL}
restart: unless-stopped
networks:
- huly_net

fulltext:
image: hardcoreeng/fulltext:${HULY_VERSION}
environment:
- SERVER_SECRET=${SECRET}
- DB_URL=${CR_DB_URL}
- FULLTEXT_DB_URL=http://elastic:9200
- ELASTIC_INDEX_NAME=huly_storage_index
- STORAGE_CONFIG=minio|minio?accessKey=minioadmin&secretKey=minioadmin
- REKONI_URL=http://rekoni:4004
- ACCOUNTS_URL=http://account:3000
- STATS_URL=http://stats:4900
- QUEUE_CONFIG=redpanda:9092
restart: unless-stopped
networks:
- huly_net

stats:
image: hardcoreeng/stats:${HULY_VERSION}
environment:
- PORT=4900
- SERVER_SECRET=${SECRET}
restart: unless-stopped
networks:
- huly_net

kvs:
image: hardcoreeng/hulykvs:${HULY_VERSION}
depends_on:
cockroach:
condition: service_started
ports:
- 8094:8094
environment:
- HULY_DB_CONNECTION=${CR_DB_URL}
- HULY_TOKEN_SECRET=${SECRET}
restart: unless-stopped
networks:
- huly_net

volumes:
# These named volumes are used when custom volume paths are not specified
# If VOLUME_*_PATH variables are set, these volumes will be ignored
elastic:
files:
cr_data:
cr_certs:
redpanda:
mongodb:

networks:
huly_net:
```