VaultDynamicSecret fails to be updated when Vault rotates the credential

[jbouse]

Describe the bug
While using a VaultDynamicSecret for a Vault database secret engine static role with a 90 day rotation period, VSO is unable to successfully rotate the k8s secret when Vault rotates the credentials.

In this instance we're rotating the database credentials for Grafana

To Reproduce
Steps to reproduce the behavior:

1. Deploy Vault static role for database connection

vault read database/static-roles/grafana                                                                                                                                       ─╯
Key                    Value
---                    -----
credential_type        password
db_name                grafana-rds
last_vault_rotation    2025-01-11T19:14:13.546015704Z
rotation_period        2160h
rotation_statements    []
username               grafana

2. Deploy Grafana with envFrom for the k8s secret to be created by VSO as required

      envFrom:
        - secretRef:
            name: grafana-db-credentials
            optional: false

3. Deploy VSO VaultDynamicSecret

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  annotations:
    database-host: <RDS DB DNS Name>:5432
  name: grafana-db-credentials
  namespace: monitoring
spec:
  allowStaticCreds: true
  destination:
    create: true
    name: grafana-db-credentials
    overwrite: true
    transformation:
      excludes:
        - password|username
      templates:
        GF_DATABASE_HOST:
          text: |
            {{- $grafana := get .Annotations "database-host" -}}
            {{ $grafana  | trim }}
        GF_DATABASE_PASSWORD:
          text: '{{ .Secrets.password | trim }}'
        GF_DATABASE_USER:
          text: '{{ .Secrets.username | trim }}'
  mount: database
  path: static-creds/grafana
  rolloutRestartTargets:
    - kind: Deployment
      name: grafana
  vaultAuthRef: grafana

The credentials are able to be retrieved manually and rotated successfully through Vault and confirmed to be operational. In fact when VSO does retrieve them and create the k8s secret it works fine. The issue is only when the credentials are rotated.

Expected behavior
Expect that when the credentials are rotated that VSO will update the credentials which should trigger the rolling restart and the application service should be able to connect to database and operate as normal.

Environment

• Kubernetes version:
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): EKS
  • Other configuration options or runtime services (istio, etc.): Vault 1.18.1
• vault-secrets-operator version: 0.9.1

[radek-sprta]

I have experienced the same behavior. A partial mitigation is to set refreshAfter to a very short interval. But that just shortens the period when application has stale credentials, rather than eliminating it.

[Aaqib041]

Hi Team,

I have also experienced same issue wherein credentials are rotated on vault but not on kubernetes secret . Currently as mitigation , I am using refreshAfter parameter in vaultDynamicSecret crd which forced CRD to refresh the secrets and also rollout the application. But this workaround is not seamless and in many scenarios the credentials may go out os sync .

Is the bug being worked on ? Any update please share