Messages not being flushed from queue on subsequent runs

message
INIT_START Runtime Version: java:8.al2.v18	Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:xxx
Transforming org/apache/logging/log4j/core/lookup/JndiLookup (lambdainternal.CustomerClassLoader@37bba400)
START RequestId: 90958b19-01b9-4744-863f-1bde7ffc5c7a Version: $LATEST
2023-03-14 06:45:00 90958b19-01b9-4744-863f-1bde7ffc5c7a INFO  GraylogS3Function:65 - Log level is now set to [DEBUG].
"io.netty.channel.socket.nio.NioSocketChannel
"
org.graylog2.gelfclient.encoder.GelfMessageJsonEncoder
"io.netty.channel.MultithreadEventLoopGroup
"
"io.netty.util.internal.InternalThreadLocalMap
"
io.netty.util.internal.PlatformDependent0
"io.netty.channel.AbstractChannel
"
"io.netty.util.concurrent.AbstractEventExecutor
"
"io.netty.util.concurrent.GlobalEventExecutor
"
org.graylog2.gelfclient.transport.AbstractGelfTransport
"io.netty.util.internal.PlatformDependent
"
org.graylog2.gelfclient.transport.GelfSenderThread
"com.github.joschi.jadconfig.JadConfig
"
"io.netty.channel.AbstractChannelHandlerContext
"
"io.netty.channel.nio.AbstractNioChannel
"
org.graylog.integrations.s3.CompressionType
"io.netty.channel.DefaultChannelId
"
"io.netty.channel.ChannelInitializer
"
"io.netty.util.concurrent.SingleThreadEventExecutor
"
"io.netty.util.NetUtilInitializations
"
org.graylog2.gelfclient.transport.GelfTcpTransport
"io.netty.util.NetUtil
"
"io.netty.util.concurrent.DefaultPromise.rejectedExecution
"
io.netty.util.internal.CleanerJava6
"io.netty.util.ResourceLeakDetector
"
"io.netty.buffer.PooledByteBufAllocator
"
"io.netty.util.internal.MacAddressUtil
"
"io.netty.util.internal.SystemPropertyUtil
"
"io.netty.channel.ChannelOutboundBuffer
"
"io.netty.util.concurrent.DefaultPromise
"
"io.netty.bootstrap.Bootstrap
"
"io.netty.resolver.AddressResolverGroup
"
"io.netty.buffer.ByteBufUtil
"
"io.netty.buffer.PoolThreadCache
"
org.graylog.integrations.s3.S3EventProcessor
"io.netty.channel.nio.NioEventLoop
"
org.graylog.integrations.s3.S3ScannerFactory
org.graylog.integrations.s3.GraylogS3Function
org.graylog.integrations.s3.ProtocolType
org.graylog.integrations.s3.ContentType
"io.netty.util.internal.logging.InternalLoggerFactory
"
"io.netty.channel.DefaultChannelPipeline
"
"io.netty.channel.ChannelHandlerMask
"
"2023-03-14 06:45:00 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GraylogS3Function:37 - Configuration{, graylogHost='xxx.xx.xx.xxx', graylogPort=12201, connectTimeout=10000, reconnectDelay=10000, tcpKeepAlive=true, tcpNoDelay=true, queueSize=4, maxInflightSends=512, contentType='text/plain', compressionType='none', protocolType='tcp', messageFields='[]', messageSummaryFields='[ClientRequestHost, ClientRequestPath, OriginIP, ClientSrcPort, EdgeServerIP, EdgeResponseBytes]'}"
2023-03-14 06:45:00 90958b19-01b9-4744-863f-1bde7ffc5c7a INFO  S3EventProcessor:45 - Reading object [prod-company-integration-webhooks-functionF19B1A04-8zoCVNRl0Zsn/2023/03/14/06-44-54-814351.log] from bucket [integration-webhooks-logs]
"2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG AbstractGelfTransport:94 - Sending message: GelfMessage{version=""1.1"" timestamp=""1678776303.922"" short_message=""[{'time': '2023-03-14T06:43:54.697Z', 'type': 'platform.start', 'record': {'requestId': 'ad89fcd8-d804-4270-bb6b-b5e816c57a22', 'version': '8'}}, {'time': '2023-03-14T06:43:54.697Z', 'type': 'function', 'record': '{""request_id"": null, ""levelname"": ""INFO"", ""name"": ""company.webhook_router.handler"", ""module"": ""handler"", ""lineno"": 97, ""message"": ""Health check PASSED"", ""timestamp"": ""2023-03-14T06:43:54.697837+00:00""}
'}, {'time': '2023-03-14T06:43:55.028Z', 'type': 'platform.end', 'record': {'requestId': 'ad89fcd8-d804-4270-bb6b-b5e816c57a22'}}, {'time': '2023-03-14T06:43:55.028Z', 'type': 'platform.report', 'record': {'requestId': 'ad89fcd8-d804-4270-bb6b-b5e816c57a22', 'metrics': {'durationMs': 331.874, 'billedDurationMs': 332, 'memorySizeMB': 128, 'maxMemoryUsedMB': 113}}}]"", level=""ALERT(1)""}"
2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a INFO  S3EventProcessor:94 - Sent [1] messages.
2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG S3EventProcessor:51 - Waiting up to [100ms] with [10] retries while waiting for transport shutdown to occur.
2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:156 - Attempting to flush messages in [100/MILLISECONDS] with [10] retries
"2023-03-14 06:45:03 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:04 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:164 - Flushing in progress. [0] messages are still enqueued, and [1] messages are still in-flight."
"2023-03-14 06:45:04 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:164 - Flushing in progress. [0] messages are still enqueued, and [1] messages are still in-flight."
"2023-03-14 06:45:04 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:164 - Flushing in progress. [0] messages are still enqueued, and [1] messages are still in-flight."
2023-03-14 06:45:04 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG GelfSenderThread:160 - Successfully flushed messages. Shutting down now.
2023-03-14 06:45:04  INFO  GelfTcpTransport:115 - Channel disconnected!
2023-03-14 06:45:04  DEBUG GelfSenderThread:123 - GelfSenderThread exiting!
2023-03-14 06:45:06  DEBUG PoolThreadCache:224 - Freed 1 thread-local buffer(s) from thread: gelfTcpTransport-1-1
2023-03-14 06:45:06 90958b19-01b9-4744-863f-1bde7ffc5c7a DEBUG S3EventProcessor:55 - Transport shutdown complete.
2023-03-14 06:45:06 90958b19-01b9-4744-863f-1bde7ffc5c7a INFO  GraylogS3Function:42 - Processed [1] S3 events.
END RequestId: 90958b19-01b9-4744-863f-1bde7ffc5c7a
REPORT RequestId: 90958b19-01b9-4744-863f-1bde7ffc5c7a	Duration: 6418.68 ms	Billed Duration: 6419 ms	Memory Size: 512 MB	Max Memory Used: 116 MB
START RequestId: d6e85b8b-e151-45c3-826c-eb5c1a5938c3 Version: $LATEST
"io.netty.channel.socket.nio.NioSocketChannel
"
org.graylog2.gelfclient.encoder.GelfMessageJsonEncoder
"io.netty.channel.MultithreadEventLoopGroup
"
"io.netty.util.internal.InternalThreadLocalMap
"
io.netty.util.internal.PlatformDependent0
"io.netty.channel.AbstractChannel
"
"io.netty.util.concurrent.AbstractEventExecutor
"
"io.netty.util.concurrent.GlobalEventExecutor
"
org.graylog2.gelfclient.transport.AbstractGelfTransport
"io.netty.util.internal.PlatformDependent
"
org.graylog2.gelfclient.transport.GelfSenderThread
"com.github.joschi.jadconfig.JadConfig
"
"io.netty.channel.AbstractChannelHandlerContext
"
"io.netty.channel.nio.AbstractNioChannel
"
org.graylog.integrations.s3.CompressionType
"io.netty.channel.DefaultChannelId
"
"io.netty.buffer.AbstractByteBuf
"
"io.netty.channel.ChannelInitializer
"
"io.netty.util.concurrent.SingleThreadEventExecutor
"
"io.netty.util.NetUtilInitializations
"
org.graylog2.gelfclient.transport.GelfTcpTransport
"io.netty.util.NetUtil
"
"io.netty.util.concurrent.DefaultPromise.rejectedExecution
"
io.netty.util.internal.CleanerJava6
"io.netty.util.ResourceLeakDetector
"
"io.netty.buffer.PooledByteBufAllocator
"
"io.netty.util.internal.MacAddressUtil
"
"io.netty.util.internal.SystemPropertyUtil
"
"io.netty.util.Recycler
"
"io.netty.channel.ChannelOutboundBuffer
"
"io.netty.util.concurrent.DefaultPromise
"
"io.netty.bootstrap.Bootstrap
"
"io.netty.resolver.AddressResolverGroup
"
"io.netty.buffer.ByteBufUtil
"
"io.netty.buffer.PoolThreadCache
"
org.graylog.integrations.s3.S3EventProcessor
"io.netty.util.ReferenceCountUtil
"
"io.netty.channel.nio.NioEventLoop
"
org.graylog.integrations.s3.S3ScannerFactory
"io.netty.util.ResourceLeakDetectorFactory
"
org.graylog.integrations.s3.GraylogS3Function
org.graylog.integrations.s3.ProtocolType
org.graylog.integrations.s3.ContentType
"io.netty.util.internal.logging.InternalLoggerFactory
"
"io.netty.channel.DefaultChannelPipeline
"
"io.netty.channel.ChannelHandlerMask
"
2023-03-14 06:45:55 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 INFO  GraylogS3Function:65 - Log level is now set to [DEBUG].
"2023-03-14 06:45:55 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GraylogS3Function:37 - Configuration{, graylogHost='xxx.xx.xx.xxx', graylogPort=12201, connectTimeout=10000, reconnectDelay=10000, tcpKeepAlive=true, tcpNoDelay=true, queueSize=4, maxInflightSends=512, contentType='text/plain', compressionType='none', protocolType='tcp', messageFields='[]', messageSummaryFields='[ClientRequestHost, ClientRequestPath, OriginIP, ClientSrcPort, EdgeServerIP, EdgeResponseBytes]'}"
2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 INFO  S3EventProcessor:45 - Reading object [prod-company-integration-webhooks-functionF19B1A04-8zoCVNRl0Zsn/2023/03/14/06-45-54-692438.log] from bucket [integration-webhooks-logs]
"2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG AbstractGelfTransport:94 - Sending message: GelfMessage{version=""1.1"" timestamp=""1678776356.731"" short_message=""[{'time': '2023-03-14T06:44:54.813Z', 'type': 'platform.start', 'record': {'requestId': 'a80f2222-c2fd-411d-9ac0-324ed2af8997', 'version': '8'}}, {'time': '2023-03-14T06:44:54.828Z', 'type': 'function', 'record': '{""request_id"": null, ""levelname"": ""INFO"", ""name"": ""company.webhook_router.handler"", ""module"": ""handler"", ""lineno"": 97, ""message"": ""Health check PASSED"", ""timestamp"": ""2023-03-14T06:44:54.814409+00:00""}
'}, {'time': '2023-03-14T06:44:55.098Z', 'type': 'platform.end', 'record': {'requestId': 'a80f2222-c2fd-411d-9ac0-324ed2af8997'}}, {'time': '2023-03-14T06:44:55.098Z', 'type': 'platform.report', 'record': {'requestId': 'a80f2222-c2fd-411d-9ac0-324ed2af8997', 'metrics': {'durationMs': 284.761, 'billedDurationMs': 285, 'memorySizeMB': 128, 'maxMemoryUsedMB': 113}}}]"", level=""ALERT(1)""}"
2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 INFO  S3EventProcessor:94 - Sent [1] messages.
2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG S3EventProcessor:51 - Waiting up to [100ms] with [10] retries while waiting for transport shutdown to occur.
2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:156 - Attempting to flush messages in [100/MILLISECONDS] with [10] retries
"2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:56 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
"2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG GelfSenderThread:164 - Flushing in progress. [1] messages are still enqueued, and [0] messages are still in-flight."
2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 ERROR GelfSenderThread:175 - Failed to flush messages in [10] attempts. Shutting down anyway.
2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 DEBUG S3EventProcessor:55 - Transport shutdown complete.
2023-03-14 06:45:57 d6e85b8b-e151-45c3-826c-eb5c1a5938c3 INFO  GraylogS3Function:42 - Processed [1] S3 events.
END RequestId: d6e85b8b-e151-45c3-826c-eb5c1a5938c3
REPORT RequestId: d6e85b8b-e151-45c3-826c-eb5c1a5938c3	Duration: 1882.47 ms	Billed Duration: 1883 ms	Memory Size: 512 MB	Max Memory Used: 116 MB