Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release-25.1: sql: improve observability of index merge timestamp #143059

Merged
merged 1 commit into from
Mar 19, 2025

Conversation

spilchen
Copy link
Contributor

Backport 1/1 commits from #142917.

/cc @cockroachdb/release


When building an index, the merge phase copies any new rows added since the index build started. A timestamp is used to set an upper bound for this scan. We suspect that the chosen timestamp may be omitting some rows.

Our theory is that this happens if new rows were inserted before the merge phase but were committed with a timestamp after the merge. This discrepancy could be due to clock skew between nodes. However, attempts to reproduce this issue via unit tests were unsuccessful, so this remains a hypothesis.

To address this, this change includes:

  • An adjustment to the merge timestamp to account for potential clock skew.
  • Additional logging of the merge timestamp chosen and the timestamps observed on each node when draining leased descriptors. These logs will help track the timestamp across nodes during the merge phase.
  • Treat index validation errors for non-unique indexes as assertion failures. We previously treated as a duplicate key error, which was very confusing because duplicates are allowed for non-unique indexes.

Epic: none
Release note: none
Closes: #142861
Closes #143050

Release justification: Bug fix for customer problem.

When building an index, the merge phase copies any new rows added since
the index build started. A timestamp is used to set an upper bound for
this scan. We suspect that the chosen timestamp may be omitting some
rows.

Our theory is that this happens if new rows were inserted before the
merge phase but were committed with a timestamp after the merge. This
discrepancy could be due to clock skew between nodes. However, attempts
to reproduce this issue via unit tests were unsuccessful, so this
remains a hypothesis.

To address this, this change includes:
- An adjustment to the merge timestamp to account for potential clock
skew.
- Additional logging of the merge timestamp chosen and the timestamps
observed on each node when draining leased descriptors. These logs will
help track the timestamp across nodes during the merge phase.
- Treat index validation errors for non-unique indexes as assertion
failures. We previously treated as a duplicat key error, which was
very confusing because duplicates are allowed for non-unique indexes.

Epic: none
Release note: none
Closes: #142751
@spilchen spilchen self-assigned this Mar 18, 2025
@spilchen spilchen requested a review from a team as a code owner March 18, 2025 12:19
Copy link

blathers-crl bot commented Mar 18, 2025

Thanks for opening a backport.

Please check the backport criteria before merging:

  • Backports should only be created for serious
    issues
    or test-only changes.
  • Backports should not break backwards-compatibility.
  • Backports should change as little code as possible.
  • Backports should not change on-disk formats or node communication protocols.
  • Backports should not add new functionality (except as defined
    here).
  • Backports must not add, edit, or otherwise modify cluster versions; or add version gates.
  • All backports must be reviewed by the owning areas TL. For more information as to how that review should be conducted, please consult the backport
    policy
    .
If your backport adds new functionality, please ensure that the following additional criteria are satisfied:
  • There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
  • The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
  • New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters. State changes must be further protected such that nodes running old binaries will not be negatively impacted by the new state (with a mixed version test added).
  • The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.
  • Your backport must be accompanied by a post to the appropriate Slack
    channel (#db-backports-point-releases or #db-backports-XX-X-release) for awareness and discussion.

Also, please add a brief release justification to the body of your PR to justify this
backport.

@blathers-crl blathers-crl bot added the backport Label PR's that are backports to older release branches label Mar 18, 2025
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@spilchen spilchen requested a review from rafiss March 18, 2025 17:58
@spilchen spilchen merged commit f29e5f9 into cockroachdb:release-25.1 Mar 19, 2025
15 of 16 checks passed
@spilchen spilchen deleted the backport25.1-142917 branch March 19, 2025 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Label PR's that are backports to older release branches target-release-25.1.4
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants