Track queued merges in ElasticsearchMergeScheduler and InternalEngine

[fcofdez]

This commit adds tracking for merges that are queued for future execution.

Relates ES-10570

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[arteam]

LGTM

[arteam]

That's an interesting idea! I guess we can piggy back on the fact that UnmodifiableCollection is just a shell and the changes to queuedMerges are guaranteed to be visible via the readOnlyQueuedMerges reference.

[kingherc]

I do see MergeTracking being used in stateless but also in stateful (InternalEngine). So I wonder why we do not call this function also for the stateful merges? An oversight? Or is it because merges cannot be queued in stateful and they just immediately execute?

If we do not mark queued merges in core ES, at least we should document it in the javadoc of markMergeQueued, hasQueuedOrOnGoingMerges() and hasQueuedMerges(), so at least people are not misled to use the functions in stateful without understanding that queued merges are not there.

[fcofdez]

Or is it because merges cannot be queued in stateful and they just immediately execute?

In stateful merges get executed immediately in their own new thread (

elasticsearch/server/src/main/java/org/elasticsearch/index/engine/ElasticsearchConcurrentMergeScheduler.java

Lines 136 to 143 in a8de554

	@Override
	protected MergeThread getMergeThread(MergeSource mergeSource, MergePolicy.OneMerge merge) throws IOException {
	MergeThread thread = super.getMergeThread(mergeSource, merge);
	thread.setName(
	EsExecutors.threadName(indexSettings, "[" + shardId.getIndexName() + "][" + shardId.id() + "]: " + thread.getName())
	);
	return thread;
	}

).

If we do not mark queued merges in core ES, at least we should document it in the javadoc of markMergeQueued, hasQueuedOrOnGoingMerges() and hasQueuedMerges(), so at least people are not misled to use the functions in stateful without understanding that queued merges are not there.

Sure, I'll add a javadoc and mention that it's implementation dependant 👍

[kingherc]

Oh OK thanks for explaining! Yea javadoc to maybe explain what you just said would be nice so that people know in stateful there are no queued merges.

[kingherc]

Oh OK thanks for explaining! Yea javadoc to maybe explain what you just said would be nice so that people know in stateful there are no queued merges.

[fcofdez]

I'm closing this in favour of a simpler solution.