[test] Fix RetrySearchIntegTests

[arteam]

Don't simultaneously restart multiple nodes in a cluster. It causes data races when multiple primaries are trying to mark the [[.snapshot-blob-cache][0]] shard as stale.

org.elasticsearch.cluster.action.shard.ShardStateAction$NoLongerPrimaryShardException: primary term [2] did not match current primary term [4]
	at org.elasticsearch.cluster.action.shard.ShardStateAction$ShardFailedClusterStateTaskExecutor.execute(ShardStateAction.java:355) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1075) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1038) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1691) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1688) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1283) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1262) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1575) ~[?:?]

Resolve #118374
Resolve #120442

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[ywangd]

I wonder whether it is worthwhile to fix the assertion in assertConsistentHistoryBetweenTranslogAndLuceneIndex to account for this edge case? Restarting nodes is something we do quite a bit in tests. So it seems reasonable to account for that. Otherwise it could fail for some other tests in future?

I am no expert on translog and lucene operations. Feel free to let me know whether the suggestion makes any sense. The change does make sense as a localised fix (to the best of my knowledge). But I am curious whether it could be more universal.

[ywangd]

Can we retain the possibility for restarting all nodes by adding ensureGreen after each restart?

[arteam]

Yeah, that was one options that I considered. I'm not sure whether it matters for the test if restart one or multiple nodes, since we just check that the search id and PIT are retained during node restarts, but I will update it to keep the original semantics of the test

[arteam]

ad8369b

[arteam]

@ywangd I'm not sure we can fix assertConsistentHistoryBetweenTranslogAndLuceneIndex because the failure is essentialy a missed failing operation on a replica shard due to a race. I think we just have to ensure that the we restart nodes step by step.

[ywangd]

LGTM

[ywangd]

I'm not sure we can fix assertConsistentHistoryBetweenTranslogAndLuceneIndex because the failure is essentialy a missed failing operation on a replica shard due to a race. I think we just have to ensure that the we restart nodes step by step.

Thanks Artem. I am good with the proposed fix. What I suggested earlier may not be viable. I am no expert on translog. Bottom line is that we can evaluate it again if similar issue happens elsewhere (if at all).

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.18	Commit could not be cherrypicked due to conflicts
❌	8.x	Commit could not be cherrypicked due to conflicts
✅	9.0

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 122919