Unable to connect to ignite with persistence enabled
- Dominant language
- Java
- Stars
- 5.1k
- Forks
- 1.9k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 46
Description
Hi
I am trying to run ignite in a cluster on kubernetes. My service tries to connect to the running ignite pod. But with ignite 2.16, the connection fails when persistence is enabled. The same operation works with persistence disabled with version 2.16 and also works with the previous versions irrespective of whether persistence is enabled or disabled.
I use thinClient to connect to ignite from my service
logsof my service -
```
org.apache.ignite.client.ClientException: Ignite failed to process request [24]: 50000: Failed to execute map query on remote node [nodeId=514c2e13-805e-4b75-8811-9b2e8343b44b, errMsg=General error: "class org.apache.ignite.binary.BinaryObjectException: Cannot find metadata for object with compact footer (Ignite work directory might have been cleared after restart. Make sure that IGNITE_HOME does not point to a temp folder or any other folder that is destroyed/cleared on restarts) [typeId=396738638, IGNITE_HOME='/opt/ignite/apache-ignite']"; SQL statement:
SELECT
PUBLIC.__Z0._KEY __C0_0,
PUBLIC.__Z0._VAL __C0_1,
__Z0.LASTUPDATED __C0_2
FROM PUBLIC.ABC __Z0
WHERE __Z0.ABCDEF = ?1
ORDER BY 3 LIMIT ?2 [50000-197]] (server status code [1])
at org.apache.ignite.internal.client.thin.TcpClientChannel.convertException(TcpClientChannel.java:475) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.TcpClientChannel.receive(TcpClientChannel.java:421) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.TcpClientChannel.service(TcpClientChannel.java:311) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ReliableChannel.lambda$service$0(ReliableChannel.java:205) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ReliableChannel.applyOnDefaultChannel(ReliableChannel.java:835) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ReliableChannel.applyOnDefaultChannel(ReliableChannel.java:788) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:205) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:186) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.GenericQueryPager.next(GenericQueryPager.java:93) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ClientQueryCursor$1.nextPage(ClientQueryCursor.java:93) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ClientQueryCursor$1.hasNext(ClientQueryCursor.java:76) ~[ignite-core-2.16.0.jar!/:2.16.0]
at java.base/java.lang.Iterable.forEach(Unknown Source) ~[na:na]
at org.apache.ignite.springdata.repository.query.IgniteRepositoryQuery$QueryCursorWrapper.getAll(IgniteRepositoryQuery.java:963) ~[ignite-spring-data-ext-2.0.0.jar!/:2.0.0]
at org.apache.ignite.springdata.repository.query.IgniteRepositoryQuery.transformQueryCursor(IgniteRepositoryQuery.java:616) ~[ignite-spring-data-ext-2.0.0.jar!/:2.0.0]
at org.apache.ignite.springdata.repository.query.IgniteRepositoryQuery.execute(IgniteRepositoryQuery.java:360) ~[ignite-spring-data-ext-2.0.0.jar!/:2.0.0]
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:136) ~[spring-data-commons-3.0.5.jar!/:3.0.5]
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:120) ~[spring-data-commons-3.0.5.jar!/:3.0.5]
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:164) ~[spring-data-commons-3.0.5.jar!/:3.0.5]
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:143) ~[spring-data-commons-3.0.5.jar!/:3.0.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:244) ~[spring-aop-6.0.11.jar!/:6.0.11]
at jdk.proxy2/jdk.proxy2.$Proxy129.getGoodABCs(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-6.0.11.jar!/:6.0.11]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:244) ~[spring-aop-6.0.11.jar!/:6.0.11]
at jdk.proxy2/jdk.proxy2.$Proxy129.getGoodABCs(Unknown Source) ~[na:na]
at ~[classes!/:na]
at reactor.core.publisher.MonoCallable.call(MonoCallable.java:72) ~[reactor-core-3.5.9.jar!/:3.5.9]
at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:127) ~[reactor-core-3.5.9.jar!/:3.5.9]
at reactor.core.publisher.MonoFlatMapMany.subscribeOrReturn(MonoFlatMapMany.java:49) ~[reactor-core-3.5.9.jar!/:3.5.9]
at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:55) ~[reactor-core-3.5.9.jar!/:3.5.9]
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) ~[reactor-core-3.5.9.jar!/:3.5.9]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.5.9.jar!/:3.5.9]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.5.9.jar!/:3.5.9]
at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
```
Contributor guide
Research direction
Start by reproducing the thin-client query against Ignite 2.16 in Kubernetes with persistence enabled, comparing it with persistence disabled and earlier versions. Read the stack-trace entry points in TcpClientChannel.java and IgniteRepositoryQuery.java, then trace the compact-footer metadata error. Done means the query succeeds with persistence enabled and the regression is covered by an appropriate test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100