apache / apache/ignite

Unable to connect to ignite with persistence enabled

Open
#11,386 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
5.1k
Forks
1.9k
Avg merge
3d 2h
Merged PRs (30d)
46

Description

Hi
I am trying to run ignite in a cluster on kubernetes. My service tries to connect to the running ignite pod. But with ignite 2.16, the connection fails when persistence is enabled. The same operation works with persistence disabled with version 2.16 and also works with the previous versions irrespective of whether persistence is enabled or disabled.
I use thinClient to connect to ignite from my service

logsof my service -
```
org.apache.ignite.client.ClientException: Ignite failed to process request [24]: 50000: Failed to execute map query on remote node [nodeId=514c2e13-805e-4b75-8811-9b2e8343b44b, errMsg=General error: "class org.apache.ignite.binary.BinaryObjectException: Cannot find metadata for object with compact footer (Ignite work directory might have been cleared after restart. Make sure that IGNITE_HOME does not point to a temp folder or any other folder that is destroyed/cleared on restarts) [typeId=396738638, IGNITE_HOME='/opt/ignite/apache-ignite']"; SQL statement:
SELECT
PUBLIC.__Z0._KEY __C0_0,
PUBLIC.__Z0._VAL __C0_1,
__Z0.LASTUPDATED __C0_2
FROM PUBLIC.ABC __Z0
WHERE __Z0.ABCDEF = ?1
ORDER BY 3 LIMIT ?2 [50000-197]] (server status code [1])
at org.apache.ignite.internal.client.thin.TcpClientChannel.convertException(TcpClientChannel.java:475) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.TcpClientChannel.receive(TcpClientChannel.java:421) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.TcpClientChannel.service(TcpClientChannel.java:311) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ReliableChannel.lambda$service$0(ReliableChannel.java:205) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ReliableChannel.applyOnDefaultChannel(ReliableChannel.java:835) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ReliableChannel.applyOnDefaultChannel(ReliableChannel.java:788) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:205) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:186) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.GenericQueryPager.next(GenericQueryPager.java:93) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ClientQueryCursor$1.nextPage(ClientQueryCursor.java:93) ~[ignite-core-2.16.0.jar!/:2.16.0]
at org.apache.ignite.internal.client.thin.ClientQueryCursor$1.hasNext(ClientQueryCursor.java:76) ~[ignite-core-2.16.0.jar!/:2.16.0]
at java.base/java.lang.Iterable.forEach(Unknown Source) ~[na:na]
at org.apache.ignite.springdata.repository.query.IgniteRepositoryQuery$QueryCursorWrapper.getAll(IgniteRepositoryQuery.java:963) ~[ignite-spring-data-ext-2.0.0.jar!/:2.0.0]
at org.apache.ignite.springdata.repository.query.IgniteRepositoryQuery.transformQueryCursor(IgniteRepositoryQuery.java:616) ~[ignite-spring-data-ext-2.0.0.jar!/:2.0.0]
at org.apache.ignite.springdata.repository.query.IgniteRepositoryQuery.execute(IgniteRepositoryQuery.java:360) ~[ignite-spring-data-ext-2.0.0.jar!/:2.0.0]
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:136) ~[spring-data-commons-3.0.5.jar!/:3.0.5]
at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:120) ~[spring-data-commons-3.0.5.jar!/:3.0.5]
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:164) ~[spring-data-commons-3.0.5.jar!/:3.0.5]
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:143) ~[spring-data-commons-3.0.5.jar!/:3.0.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:244) ~[spring-aop-6.0.11.jar!/:6.0.11]
at jdk.proxy2/jdk.proxy2.$Proxy129.getGoodABCs(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-6.0.11.jar!/:6.0.11]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.0.11.jar!/:6.0.11]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:244) ~[spring-aop-6.0.11.jar!/:6.0.11]
at jdk.proxy2/jdk.proxy2.$Proxy129.getGoodABCs(Unknown Source) ~[na:na]
at ~[classes!/:na]
at reactor.core.publisher.MonoCallable.call(MonoCallable.java:72) ~[reactor-core-3.5.9.jar!/:3.5.9]
at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:127) ~[reactor-core-3.5.9.jar!/:3.5.9]
at reactor.core.publisher.MonoFlatMapMany.subscribeOrReturn(MonoFlatMapMany.java:49) ~[reactor-core-3.5.9.jar!/:3.5.9]
at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:55) ~[reactor-core-3.5.9.jar!/:3.5.9]
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) ~[reactor-core-3.5.9.jar!/:3.5.9]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.5.9.jar!/:3.5.9]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.5.9.jar!/:3.5.9]
at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the thin-client query against Ignite 2.16 in Kubernetes with persistence enabled, comparing it with persistence disabled and earlier versions. Read the stack-trace entry points in TcpClientChannel.java and IgniteRepositoryQuery.java, then trace the compact-footer metadata error. Done means the query succeeds with persistence enabled and the regression is covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.