elastic / elastic/logstash

Logstash doesn't spin up but raises `java.io.IOException: Checkpoint checksum mismatch, expected: 278197, actual: 0` error.

Open
#16,389 2 comments 0 reactions 1 assignee Claimed by @mashhurs View on GitHub
bug status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

**Logstash information**: 8.14.1

Please include the following information:

**The root cause for this exception is `checkpoint.head` file somehow corrupted. The quick remediation is to overwrite `checkpoint.head` with `checkpoint.head.tmp` (if available).**

1. Logstash version (e.g. `bin/logstash --version`)
2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)

**Plugins installed**: (`bin/logstash-plugin list --verbose`) default

**JVM** (e.g. `java -version`): bundled

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

1. JVM version (`java -version`)
2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
3. Value of the `LS_JAVA_HOME` environment variable if set.

**OS version** (`uname -a` if on a Unix-like system):

**Description of the problem including expected versus actual behavior**:

**Steps to reproduce**:

I failed to reproduce the issue. From the error we can see the disk is full but when disk is full we get `No space left on device` when LS is starting: https://github.com/elastic/logstash/issues/16389#issuecomment-2302656261

**Provide logs (if relevant)**:
- Behavioral log
```
The persistent queue on path "/usr/share/logstash/data/queue/logging" won't fit in file system "/dev/mapper/lxc-data" when full. Please free or allocate 19814579200 more bytes. The persistent queue on path "/usr/share/logstash/data/queue/logging_metering" won't fit in file system "/dev/mapper/lxc-data" when full. Please free or allocate 19814579200 more bytes. The persistent queue on path "/usr/share/logstash/data/queue/logging_metering_snapshot_estimator" won't fit in file system "/dev/mapper/lxc-data" when full. Please free or allocate 19814579200 more bytes. The persistent queue on path "/usr/share/logstash/data/queue/logging_service" won't fit in file system "/dev/mapper/lxc-data" when full. Please free or allocate 19814579200 more bytes. The persistent queue on path "/usr/share/logstash/data/queue/auditrecord_logstash" won't fit in file system "/dev/mapper/lxc-data" when full. Please free or allocate 19814579200 more bytes.
abcd.centralus.azure.elastic-cloud.com:9243 failed to respond
Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://abcd.centralus.azure.elastic-cloud.com:9243/_bulk?filter_path=errors,items.*.error,items.*.status][Manticore::ClientProtocolException] abcd.centralus.azure.elastic-cloud.com:9243 failed to respond
Elasticsearch Unreachable: [https://abcd.centralus.azure.elastic-cloud.com:9243/_bulk?filter_path=errors,items.*.error,items.*.status][Manticore::ClientProtocolException] abcd.centralus.azure.elastic-cloud.com:9243 failed to respond

Could not index event to Elasticsearch.
Could not index event to Elasticsearch.
Could not index event to Elasticsearch.
Could not index event to Elasticsearch.

Starting Logstash
...
Logstash failed to create queue.
-----> java.io.IOException: Checkpoint checksum mismatch, expected: 278197, actual: 0
```

- Stack trace
```
{"level":"INFO","loggerName":"logstash.runner","timeMillis":1721353466915,"thread":"main","logEvent":{"message":"Starting Logstash","logstash.version":"8.14.1","jruby.version":"jruby 9.4.7.0 (3.1.4) 2024-04-29 597ff08ac1 OpenJDK 64-Bit Server VM 17.0.11+9 on 17.0.11+9 +indy +jit [x86_64-linux]"}}
{
"level": "ERROR",
"loggerName": "logstash.agent",
"timeMillis": 1722887281351,
"thread": "Converge PipelineAction::Create",
"logEvent": {
"message": "java.io.IOException: Checkpoint checksum mismatch, expected: 278197, actual: 0",
"action": "LogStash::PipelineAction::Create/pipeline_id:logging",
"exception": "Java::JavaLang::IllegalStateException",
"backtrace": [
"org.logstash.execution.AbstractPipelineExt.openQueue(AbstractPipelineExt.java:260)",
"org.logstash.execution.AbstractPipelineExt$INVOKER$i$0$0$openQueue.call(AbstractPipelineExt$INVOKER$i$0$0$openQueue.gen)",
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:456)",
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)",
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:346)",
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)",
"org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:94)",
"org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:275)",
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:262)",
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:236)",
"usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49)",
"usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386)",
"org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)",
"org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)",
"org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)",
"org.jruby.runtime.Block.call(Block.java:144)",
"org.jruby.RubyProc.call(RubyProc.java:354)",
"org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)",
"java.base/java.lang.Thread.run(Thread.java:840)"
]
}
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.