aws / aws/aws-xray-sdk-java

Workarounds and visibility on IOException: Message too long

Open
#26 10 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
100
Forks
100
PR merge metrics
No merged PRs in 30d

Description

We are running into the well known (#4) problem that sometimes a segment is too big to fit in a UDP message (65,507 bytes). For example because it contains a big sql statement or an exception.

The problem is however that if these issues happen sporadically they are very difficult to debug as there is almost no visibility.

We see a few possible improvements which can be made easily:

* _Enhance visibility_: For example by logging the segment when it is too big.
It could be as easy as doing something like this:
```diff
}

- private boolean sendData(byte[] data) {
+ private boolean sendData(byte[] data, Entity entity) {
+ if (data.length > 65_527) {
+ logger.warn("(sub)segment is too big to send: " + entity.prettySerialize());
+ return false;
+ }
DatagramPacket packet = new DatagramPacket(sendBuffer, DAEMON_BUF_RECEIVE_SIZE, config.address);
```
(Btw: it also puzzles us why `DAEMON_BUF_RECEIVE_SIZE` is a lot bigger then 65_527)

* _Add retry behavior_ which tries to send the completed subsegments (so if one is too big, it does not break the others). We for example currently quickly implemented a `RetryingEmitter` which works like this:

```java
public class RetryingEmitter extends UDPEmitter {
private static final Log logger = LogFactory.getLog(MethodHandles.lookup().lookupClass());
private final StreamingStrategy streamingStrategy;
public RetryingEmitter(StreamingStrategy streamingStrategy) throws SocketException {
super();
this.streamingStrategy = streamingStrategy;
}
public boolean sendSegment(Segment segment) {return trySplitOnFailure(segment, super::sendSegment);}
public boolean sendSubsegment(Subsegment subsegment) {return trySplitOnFailure(subsegment, super::sendSubsegment);}
boolean trySplitOnFailure(T entity, Predicate send) {
boolean ok = send.test(entity);
if (!ok) {
logger.info("send of entity " + entity.getId() + " failed, retrying.");
streamingStrategy.streamSome(entity, this);
ok = send.test(entity);
if (!ok) {
logger.warn("Unable to send entity " + entity.getId() + " after split. size: " +
entity.serialize().length() + ", content: " + entity.prettySerialize());
}
}
return ok;
}
}
```
This is of course rather naive, as a lot of errors are still send. But it helps us to debug these kinds of issues.

What do you think about these proposals?

Thanks,

Steven

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.