Mark volume as uncertain if MarkVolumeAsAttached fails #129664

hkttty2009 · 2025-01-16T15:05:40Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

After volumeAttacher.Attach executes successfully, the volumeattachment will be created. At this point, if CSI's ATTACHREQUIRED changes from true to false, it will cause MarkVolumeAsAttached to fail.
And the volume will be removed from the dsw, leaving the VolumeAttachment stranded without being properly cleaned up.
In this PR, we mark the volume as uncertain to allow volumeattachment cleanup on the next retry.

Which issue(s) this PR fixes:

Fixes #129572

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fixed volumeattachment cleanup in kube-controller-manager when CSI's attachRequired switches from true to false

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

When CSI's attachRequired changes from true to false after successful volume attach, MarkVolumeAsAttached may fail, leaving VolumeAttachment stranded. Mark volume as uncertain to allow volumeattachment cleanup on next retry. Signed-off-by: hongkang <[email protected]>

linux-foundation-easycla · 2025-01-16T15:05:45Z

The committers listed above are authorized under a signed CLA.

✅ login: hkttty2009 / name: hongkang (b367b30, 47a940c)

k8s-ci-robot · 2025-01-16T15:05:49Z

Welcome @hkttty2009!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-01-16T15:05:49Z

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-01-16T15:05:50Z

Hi @hkttty2009. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

carlory · 2025-01-17T02:37:38Z

Please add a release note because it is bugfix and it should introduce a user-facing change. And is it possible to add an e2e test for this PR to verify the bugfix? Thanks for the contribution!
/ok-to-test

k8s-ci-robot · 2025-01-19T17:30:52Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hkttty2009
Once this PR has been reviewed and has the lgtm label, please assign saad-ali for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hkttty2009 · 2025-01-20T02:23:16Z

Please add a release note because it is bugfix and it should introduce a user-facing change. And is it possible to add an e2e test for this PR to verify the bugfix? Thanks for the contribution! /ok-to-test

I've added a release note and e2e test. Please review the updates,thanks!

hkttty2009 · 2025-02-08T11:07:31Z

Hi @carlory, could you please review this PR? Thank you for your time and feedback !

carlory · 2025-02-08T12:19:26Z

sorry for the delay，I will review it next week. It is in my todo list.

carlory · 2025-02-13T09:05:34Z

test/e2e/storage/csimock/csi_attach_volume.go

+			ginkgo.DeferCleanup(func(ctx context.Context) {
+				_, err := m.cs.StorageV1().VolumeAttachments().Get(ctx, attachmentName, metav1.GetOptions{})
+				if err == nil {
+					err := m.cs.StorageV1().VolumeAttachments().Delete(ctx, attachmentName, metav1.DeleteOptions{})
+					framework.ExpectNoError(err, "Failed to delete VolumeAttachment: %v", err)
+				} else if !apierrors.IsNotFound(err) {
+					framework.ExpectNoError(err, "Failed to get VolumeAttachment")
+				}
+			})
+


It makes code simpler.

Suggested change

ginkgo.DeferCleanup(func(ctx context.Context) {

_, err := m.cs.StorageV1().VolumeAttachments().Get(ctx, attachmentName, metav1.GetOptions{})

if err == nil {

err := m.cs.StorageV1().VolumeAttachments().Delete(ctx, attachmentName, metav1.DeleteOptions{})

framework.ExpectNoError(err, "Failed to delete VolumeAttachment: %v", err)

} else if !apierrors.IsNotFound(err) {

framework.ExpectNoError(err, "Failed to get VolumeAttachment")

}

})

ginkgo.DeferCleanup(framework.IgnoreNotFound(m.cs.StorageV1().VolumeAttachments().Delete), attachmentName, metav1.DeleteOptions{})

carlory · 2025-02-13T09:30:22Z

test/e2e/storage/csimock/csi_attach_volume.go

+			framework.ExpectNoError(err, "Failed to get CSIDriver: %v", err)
+
+			ginkgo.By("Wait for the volumeattachment to be deleted")
+			err = e2evolume.WaitForVolumeAttachmentTerminated(ctx, attachmentName, m.cs, csiVolumeAttachmentTimeout)


If the csi driver doesn't deploy the attacher component, the expected status is terminating, not terminated once the finalizer is added to the volume attachment. In this case, we have to unset finalizer manually in order to cleanup this resource later. The new e2e test does not redeloyment the whole csi driver with disableAttach: true, only changing the csi driver attributes, so the volume attachment can be deleted from kube-apiserver. It looks good to me.

Thank you for the comment. As you mentioned, retaining the attacher component allows the VolumeAttachment to be cleaned up from kube-apiserver. I'm currently developing a CSI driver that changed its attachRequired from true to false but still retains the attacher component after this change. So I think retaining the attacher component aligns more with real-world scenarios.

@hkttty2009 Does the controller capabilities of your csi driver support the ControllerServiceCapability_RPC_PUBLISH_UNPUBLISH_VOLUME capability when you change the ATTACHREQUIRED field from true to false?

If not supported, the attacher won't remove the finalizer, please see https://github.com/kubernetes-csi/external-attacher/blob/master/pkg/controller/trivial_handler.go#L50

If supported, why change the ATTACHREQUIRED to false?

Our CSI driver support the ControllerServiceCapability_RPC_PUBLISH_UNPUBLISH_VOLUME capability, but in reality, the CSI's ControllerPublishVolume doesn't perform any operations.
This is an NFS CSI driver which doesn't actually require attachment, but for a long period of time, the CSI's ATTACHREQUIRED was set to true. Changing it to false is to speed up pod creation by eliminating the step of creating volumeattachment.
I discovered the VolumeAttachment cleanup issue during CSI upgrade validation scenarios, which led to submitting this PR to address the problem.

carlory · 2025-02-13T10:07:41Z

/cc @saad-ali
Can you take another look? Thank you.

…ed changes Signed-off-by: hongkang <[email protected]>

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jan 16, 2025

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 16, 2025

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 16, 2025

k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 16, 2025

k8s-ci-robot requested review from carlory and saikat-royc January 16, 2025 15:06

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jan 16, 2025

hkttty2009 force-pushed the bug/volumeattachment branch 2 times, most recently from a443f93 to c135b59 Compare January 19, 2025 18:21

carlory reviewed Feb 13, 2025

View reviewed changes

k8s-ci-robot requested a review from saad-ali February 13, 2025 10:07

Add e2e test for VolumeAttachment cleanup when CSIDriver AttachRequir…

b367b30

…ed changes Signed-off-by: hongkang <[email protected]>

hkttty2009 force-pushed the bug/volumeattachment branch from c135b59 to b367b30 Compare February 14, 2025 07:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mark volume as uncertain if MarkVolumeAsAttached fails #129664

Mark volume as uncertain if MarkVolumeAsAttached fails #129664

hkttty2009 commented Jan 16, 2025 •

edited

Loading

linux-foundation-easycla bot commented Jan 16, 2025 •

edited

Loading

k8s-ci-robot commented Jan 16, 2025

k8s-ci-robot commented Jan 16, 2025

k8s-ci-robot commented Jan 16, 2025

carlory commented Jan 17, 2025

k8s-ci-robot commented Jan 19, 2025

hkttty2009 commented Jan 20, 2025

hkttty2009 commented Feb 8, 2025

carlory commented Feb 8, 2025

carlory Feb 13, 2025

hkttty2009 Feb 20, 2025

carlory Feb 13, 2025

hkttty2009 Feb 20, 2025

carlory Feb 26, 2025

carlory Feb 26, 2025

carlory Feb 26, 2025

hkttty2009 Feb 26, 2025

carlory commented Feb 13, 2025

Mark volume as uncertain if MarkVolumeAsAttached fails #129664

Are you sure you want to change the base?

Mark volume as uncertain if MarkVolumeAsAttached fails #129664

Conversation

hkttty2009 commented Jan 16, 2025 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

linux-foundation-easycla bot commented Jan 16, 2025 • edited Loading

k8s-ci-robot commented Jan 16, 2025

k8s-ci-robot commented Jan 16, 2025

k8s-ci-robot commented Jan 16, 2025

carlory commented Jan 17, 2025

k8s-ci-robot commented Jan 19, 2025

hkttty2009 commented Jan 20, 2025

hkttty2009 commented Feb 8, 2025

carlory commented Feb 8, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlory commented Feb 13, 2025

hkttty2009 commented Jan 16, 2025 •

edited

Loading

linux-foundation-easycla bot commented Jan 16, 2025 •

edited

Loading