Apache Ignite - 자동화된 클러스터 스냅샷 생성/복원 프로세스

Apache Ignite 클러스터 스냅샷 프로세스(스냅샷 생성, 복원)를 시도할 때 복원 작업이 실패합니다.

복원 프로세스 전에 공식 문서에 따라 복원할 캐시를 삭제했지만 동일한 오류 메시지/Java 예외가 여전히 로그에 존재합니다.

Apache Ignite - 스냅샷 생성/복원 프로세스:

./bin/control.sh --snapshot create 20230412_snapshot_test_v1
./bin/control.sh --cache destroy --caches CACHE_XYZ --yes
./bin/control.sh --snapshot restore 20230412_snapshot_test_v1 --start --groups CACHE_XYZ

Apache Ignite - 로그:

The snapshot operation will be aborted due to a handler error 
class org.apache.ignite.IgniteCheckedException: The check procedure failed on 3 nodes.

The check procedure failed on nodes:

Node ID: 260115a7-95b4-48a3-bc98-af9d8af0df51 [192.168.XX.YY, 127.0.0.1]
Consistent ID: 547359f6-4501-4727-8563-a8b170a905f3
Exception: java.lang.NegativeArraySizeException
-935925367

Node ID: 33d84ca7-e4e2-4b6e-9c3c-63625c853dc1 [192.168.XX.YY, 127.0.0.1]
Consistent ID: aa7da35d-37b3-401d-8df3-f3c6e2761c85
Exception: java.lang.NegativeArraySizeException
-935925367

Node ID: d512f146-abb2-4e88-a22c-3a640e84be18 [192.168.XX.YY, 127.0.0.1]
Consistent ID: 0b6fafa5-a31f-4fa7-80c5-44c792af03ff
Exception: java.lang.NegativeArraySizeException
-935925367

...

Failed to restore snapshot cache groups 
class org.apache.ignite.compute.ComputeUserUndeclaredException: Failed to reduce job results due to undeclared user exception [task=org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotHandlerRestoreTask@4c606da9, err=class org.apache.ignite.IgniteException: The check procedure failed on 3 nodes.

또 다른 이상한 점은 어떤 이유로 --snapshot check클러스터에 연결할 수 없지만 다른 명령은 모두 잘 작동한다는 것입니다.

Apache Ignite - 클러스터 스냅샷 확인:

./bin/control.sh --snapshot check 20230412_snapshot_test_v1 --verbose

...

INFO: Client TCP connection established: /127.0.0.1:11211

...

SEVERE: Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.DirectByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-client-0, igniteInstanceName=gridClient, finished=false, heartbeatTs=1682407593249, hashCode=245934772, interrupted=false, runner=grid-nio-worker-tcp-client-0-#5%gridClient%]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, closeSocket=true, outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:42122, rmtAddr=/127.0.0.1:11211, createTime=1682407593249, closeTime=0, bytesSent=702, bytesRcvd=78929, bytesSent0=702, bytesRcvd0=78929, sndSchedTime=1682407593249, lastSndTime=1682407593249, lastRcvTime=1682407593249, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=GridTcpRestParser [marsh=JdkMarshaller [clsFilter=null], routerClient=false], directMode=false]], accepted=false, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
        at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1183)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2508)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2273)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1910)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
        at java.lang.Thread.run(Thread.java:750)

...

class org.apache.ignite.internal.client.impl.connection.GridClientConnectionResetException: Failed to perform request (connection failed): /127.0.0.1:11211

Apache Ignite - 로그:

[SnapshotPartitionsVerifyTask$VisorVerifySnapshotPartitionsJob] Verify snapshot partitions procedure has been initiated [snpName=20230412_snapshot_test_v1, consId=33303d92-f3db-4b33-be38-798167c2f726]
[IgniteSnapshotManager] Resolved directory for serialized binary metadata: /usr/share/apache-ignite/work/snapshots/20230412_snapshot_test_v1/db/binary_meta/node00-33303d92-f3db-4b33-be38-798167c2f726

... nothing else is printed, the service goes down ...

스냅샷 확인이 실패한 직후(위에 설명된 연결 오류) 각 Linux 서버의 Apache Ignite 서비스가 중지되고 클러스터가 종료되며 모든 것을 수동으로 다시 시작해야 합니다.

이는 Apache ZooKeeper로 구성된 3노드 Apache Ignite 클러스터입니다(각 노드는 서로 다른 서버에 있음). OpenJDK 1.8.0을 사용하여 Linux에서 실행되는 Apache Ignite 2.13.0 및 2.14.0에서도 동일한 문제가 발생합니다.

복구 프로세스가 실패한 이유는 무엇입니까?