kvserver/closedts: auto-tune closed ts

[wenyihu6]

kvserver/closedts: add latency based closed timestamp policies

This patch adds latency based closed timestamp policies to
ctpb.LatencyBasedRangeClosedTimestampPolicy.

Part of: #59680
Release note: none

---

kvserver/closedts: auto-tune closed ts

For global table ranges, we aim to ensure follower replicas can serve
present-time reads by calculating a target closed timestamp that leads the
current time (using the LEAD_FOR_GLOBAL_READS policy). The goal of this
estimation is to be confident that by that timestamp that the all follower
replicas should have received this closed timestamps and can serve present time
reads without redirecting to the leaseholders. Note that this is an estimated
goal, not the exact lead time. It is still possible that some follower nodes are
really behind or we fail to bump closed timestamps.

Calculation:

raft_propagation_time = max_network_rtt*1.5 + raft_overhead(20ms)
side_propagation_time = max_network_rtt*0.5 + side_transport_close_interval(200ms by default)
closed_ts_propagation = max(raft_propagation_time,side_propagation_time)
closed_ts_at_sender = now(sender’s current clock) + maxClockOffset(500ms by
default) + buffer (25ms) + closed_ts_propagation

Currently, max_network_rtt is hardcoded at 150ms, leading to a default 800ms lead.

Problems with the current implementation:

Tuning trade-offs: A higher lead timestamp forces leaseholders to wait longer
for the current time to catch up before returning to client, incurring a high
write latency. A lower lead timestamp may result in followers not fully
replicating the changes before returning control to the client, forcing follower
replicas to redirect query to the leaseholder and increase read latency.

Cluster wide setting: kv.closed_timestamp.lead_for_global_reads_override only
applies to the entire cluster, not per range. For geographically distant
replicas, a short lead time may not be enough for full application. However,
increasing the lead time cluster wide can harm write performance for replicas
that are close together and don’t require as much time to replicate.

This commit adds a latency tracker to the side transport senders.

Currently, nodes create a map of closed timestamps for each policy without
consulting leaseholders. Senders pass this map to leaseholders, who decide
whether to "bump" their closed timestamp and which policy to use. Senders then
send this information to receivers to opt in ranges and pick the closed
timestamps based on the policies ranges opt in.

Currently there are only two policies in the map side transport senders create.
We lack fine grained control into the closed timestamps based on the ranges. The
new design adds 16 more policies based on observed latency between the leaseholder
replica node and the furthest replica node.

Periodically, policy refresher iterates over all leaseholders of a node and
refreshes its cached closed timestamp policies. This policy is then used by
side transport and by raft side closed timestamp computation.

kv.closed_timestamp.lead_for_global_reads_auto_tune.enabled can now be used
to auto-tune the lead time that ranges global tables use to publish
close timestamps. The kv.closed_timestamp.lead_for_global_reads_override
cluster setting takes precedence over this one. If auto-tuning is disabled or
no data is observed, the system falls back to a hardcoded computed lead time.

Resolves: #59680
Release note: none

[blathers-crl]

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is