apache / apache/pulsar-manager

fail to create environment as broker throw error: `io.netty.handler.codec.TooLongFrameException`

Open
#473 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Vue
Stars
539
Forks
239
PR merge metrics
No merged PRs in 30d

Description

when i create new environment, got error: `This environment is error. Please check it`

container log:
```log
pulsar_1 | 2022-07-19T04:57:04,910+0000 [pulsar-io-31-4] WARN org.apache.pulsar.broker.service.ServerCnx - [/172.31.0.3:40930] Got exception io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5253120: 1195725860 - discarded
pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:507)
pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:493)
pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:377)
pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:423)
pulsar_1 | at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:333)
pulsar_1 | at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
pulsar_1 | at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
pulsar_1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
pulsar_1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
pulsar_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
pulsar_1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
pulsar_1 | at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
pulsar_1 | at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
pulsar_1 | at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
pulsar_1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
pulsar_1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
pulsar_1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
pulsar_1 | at java.base/java.lang.Thread.run(Thread.java:829)
```

i found this issue: https://github.com/apache/pulsar/issues/15688

```conf
# The maximum netty frame size in bytes. Any message received larger than this will be rejected. The default value is 5MB.
nettyMaxFrameSizeBytes=5253120

5,253,120
1,195,725,860 # 1G ??

```
is manager backend call the wrong interface? why the data size is 1G?
this is our test environment with 100 topic and not too much msgs.

https://github.com/apache/pulsar-manager/blob/master/src/main/java/org/apache/pulsar/manager/controller/EnvironmentsController.java#L124
```java
pulsarAdminService.clusters(environmentEntity.getBroker()).getClusters();

```

i test locally with no topic, got the same error: ` Adjusted frame length exceeds 5253120: 1195725860`
same data size.

my configuration:
```yaml
# docker-compose.yml
---
version: "3.7"
services:
pulsar:
image: apachepulsar/pulsar:2.10.1
environment:
- PULSAR_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=1g
- PULSAR_PREFIX_defaultRetentionTimeInMinutes=14400
- PULSAR_PREFIX_defaultRetentionSizeInMB=1000
# - PULSAR_PREFIX_httpServerEnabled=true
- PULSAR_PREFIX_bookkeeperClientExposeStatsToPrometheus=true
- PULSAR_PREFIX_transactionCoordinatorEnabled=true
# - PULSAR_PREFIX_nettyMaxFrameSizeBytes=5,253,120 # default
# - PULSAR_PREFIX_nettyMaxFrameSizeBytes=1,195,725,860
# command: bin/pulsar standalone
command: >
/bin/bash -c
"bin/apply-config-from-env.py conf/standalone.conf && bin/pulsar standalone"
hostname: pulsar
ports:
- "8090:8080"
- "6650:6650"
restart: unless-stopped
volumes:
- "./data/:/pulsar/data"
dashboard:
image: apachepulsar/pulsar-manager:v0.3.0
ports:
- "8091:9527"
- "7750:7750"
depends_on:
- pulsar
links:
- pulsar
environment:
SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
USERNAME: "pulsar"
PASSWORD: "pulsar"
...
```

Contributor guide

Open the contributing guide

Research direction

Start at src/main/java/org/apache/pulsar/manager/controller/EnvironmentsController.java:124 and reproduce environment creation with the provided docker-compose.yml and Pulsar 2.10.1 configuration. Trace the clusters(environmentEntity.getBroker()).getClusters() call and compare it with the reported Netty frame-size error. Done means environment creation completes without the TooLongFrameException and the behavior is verified in the local no-topic setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.