apache / apache/pulsar-manager
fail to create environment as broker throw error: `io.netty.handler.codec.TooLongFrameException`
- Dominant language
- Vue
- Stars
- 539
- Forks
- 239
- PR merge metrics
- No merged PRs in 30d
Description
when i create new environment, got error: `This environment is error. Please check it`
container log:
```log
pulsar_1 | 2022-07-19T04:57:04,910+0000 [pulsar-io-31-4] WARN org.apache.pulsar.broker.service.ServerCnx - [/172.31.0.3:40930] Got exception io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5253120: 1195725860 - discarded
pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:507)
pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:493)
pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:377)
pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:423)
pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:333)
pulsar_1 | at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
pulsar_1 | at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
pulsar_1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
pulsar_1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
pulsar_1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
pulsar_1 | at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
pulsar_1 | at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
pulsar_1 | at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
pulsar_1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
pulsar_1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
pulsar_1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
pulsar_1 | at java.base/java.lang.Thread.run(Thread.java:829)
```
i found this issue: https://github.com/apache/pulsar/issues/15688
```conf
# The maximum netty frame size in bytes. Any message received larger than this will be rejected. The default value is 5MB.
nettyMaxFrameSizeBytes=5253120
5,253,120
1,195,725,860 # 1G ??
```
is manager backend call the wrong interface? why the data size is 1G?
this is our test environment with 100 topic and not too much msgs.
https://github.com/apache/pulsar-manager/blob/master/src/main/java/org/apache/pulsar/manager/controller/EnvironmentsController.java#L124
```java
pulsarAdminService.clusters(environmentEntity.getBroker()).getClusters();
```
i test locally with no topic, got the same error: ` Adjusted frame length exceeds 5253120: 1195725860`
same data size.
my configuration:
```yaml
# docker-compose.yml
---
version: "3.7"
services:
pulsar:
image: apachepulsar/pulsar:2.10.1
environment:
- PULSAR_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=1g
- PULSAR_PREFIX_defaultRetentionTimeInMinutes=14400
- PULSAR_PREFIX_defaultRetentionSizeInMB=1000
# - PULSAR_PREFIX_httpServerEnabled=true
- PULSAR_PREFIX_bookkeeperClientExposeStatsToPrometheus=true
- PULSAR_PREFIX_transactionCoordinatorEnabled=true
# - PULSAR_PREFIX_nettyMaxFrameSizeBytes=5,253,120 # default
# - PULSAR_PREFIX_nettyMaxFrameSizeBytes=1,195,725,860
# command: bin/pulsar standalone
command: >
/bin/bash -c
"bin/apply-config-from-env.py conf/standalone.conf && bin/pulsar standalone"
hostname: pulsar
ports:
- "8090:8080"
- "6650:6650"
restart: unless-stopped
volumes:
- "./data/:/pulsar/data"
dashboard:
image: apachepulsar/pulsar-manager:v0.3.0
ports:
- "8091:9527"
- "7750:7750"
depends_on:
- pulsar
links:
- pulsar
environment:
SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
USERNAME: "pulsar"
PASSWORD: "pulsar"
...
```
Contributor guide
Research direction
Start at src/main/java/org/apache/pulsar/manager/controller/EnvironmentsController.java:124 and reproduce environment creation with the provided docker-compose.yml and Pulsar 2.10.1 configuration. Trace the clusters(environmentEntity.getBroker()).getClusters() call and compare it with the reported Netty frame-size error. Done means environment creation completes without the TooLongFrameException and the behavior is verified in the local no-topic setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100