splunk / splunk/splunk-library-javalogging

Logback Appender and message size

Open
#271 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
140
Forks
173
PR merge metrics
No merged PRs in 30d

Description

Hi all,

we are using HttpEventCollectorLogbackAppender in raw mode and it looks like sent messages are somehow cut at a bit less than 10240 bytes.
Checking source code I see there is a limit in batchSize (coincidential?) at 10240 and I cannot override it as we are in raw mode[1]: is there any tip around this issue? Sometime we need to send large-ish json payload in the message and we are losing informations.
This is our logback.xml configuration:

			<appender name="SPLUNK_APPENDER"
				class="com.splunk.logging.HttpEventCollectorLogbackAppender">
				<url>${SPLUNK_URL}</url>
				<token>${SPLUNK_TOKEN}</token>
				<index>${SPLUNK_INDEX}</index>
				<!-- disableCertificateValidation should be true as we do not use valid certificates -->
				<disableCertificateValidation>true</disableCertificateValidation>
				
				<source>${SOURCE}</source>
				<sourcetype>logback</sourcetype>
				<messageFormat>text</messageFormat>
				<type>raw</type>
				<layout class="ch.qos.logback.classic.PatternLayout">
					<pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX, UTC} %.-3level SPAN_ID=%X{span_id} TRACE_ID=%X{trace_id} callId=%X{CALL_ID} uID=%X{USER_ID} ST=%X{SESSION_TOKEN} CH=%X{CHANNEL} %logger{35} - %msg%n%xException</pattern>
				</layout>
			</appender>

Thanks in advance for any help

[1]https://github.com/splunk/splunk-library-javalogging/blob/cf59a50d63ef495398f30a8f59b96b5bb62962d5/src/main/java/com/splunk/logging/HttpEventCollectorLogbackAppender.java#L251

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/main/java/com/splunk/logging/HttpEventCollectorLogbackAppender.java at the batchSize limit referenced in the issue, then trace how raw mode handles the configured Logback message. Reproduce the behavior with the supplied logback.xml and a JSON payload larger than 10240 bytes. Done means the supported behavior for large raw messages is clear and the payload is no longer unexpectedly truncated, or an explicit workaround is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.