High Availability does not work with k3s native Kine implementation

[paalkr]

What happened?

k3s natively supports Kine for using external datatores, like PostgreSQL. https://docs.k3s.io/datastore
Utilizing this feature by adding the K3S_DATASTORE_ENDPOINT environmental variable does not work with HA. See config snippet below.

Running with a single pod (controlPlane.statefulSet.highAvailability.replicas: 1) works perfectly fine, and the single pod connects to the PostgreSQL databases as expected. But running with 3 replicas does not work. This error message is thrown by the vcluster pods.

syncer 2025-02-12 13:03:17    ERROR    cmd/root.go:50    error    {"component": "vcluster", "error": "embedded database is not supported with multiple replicas"}

But as shown in the config snippet below all internal and external databases are disabled. They are not needed because k3s actually uses the connection specified in the K3S_DATASTORE_ENDPOINT environmental variable.

What did you expect to happen?

I would expect vcluster to not throw an error message and multiple pods to be successfully lunched if specifying several replicas.

How can we reproduce it (as minimally and precisely as possible)?

Look at the config snippet below. All code needed to replicate is present.

Anything else we need to know?

With vcluster version 0.19.x using HA, k3s and the K3S_DATASTORE_ENDPOINT environmental variable works without issues.

Host cluster Kubernetes version

Major:"1", Minor:"31", GitVersion:"v1.31.5-eks-8cce635"

vcluster version

0.22.4

VCluster Config

helm values snippet

controlPlane:
  distro:
    k3s:
      enabled: true
      image:
        repository: rancher/k3s
        tag: v1.31.4-k3s1
      imagePullPolicy: "IfNotPresent"
      extraArgs:
        - --kube-apiserver-arg=oidc-issuer-url=https://login.microsoftonline.com/xxx/v2.0
        - --kube-apiserver-arg=oidc-client-id=xxx
        - --kube-apiserver-arg=oidc-username-claim=email
        - --kube-apiserver-arg=oidc-username-prefix=oidc-
        - --kube-apiserver-arg=oidc-groups-claim=groups
        - --kube-apiserver-arg=oidc-groups-prefix=oidc-g-
      resources:
        limits:
          cpu: 2
          memory: 512Mi
        requests:
          cpu: 20m
          memory: 256Mi
  statefulSet:
    highAvailability:
      replicas: 3
    image:
      registry: "ghcr.io"
      repository: "loft-sh/vcluster-oss"
    imagePullPolicy: "IfNotPresent"
    env:
    - name: K3S_DATASTORE_ENDPOINT
      valueFrom:
        secretKeyRef:
          key: K3S_DATASTORE_ENDPOINT
          name: k3s-state-store-connection
    persistence:
      volumeClaim:
        enabled: false
  backingStore:
    database:
      embedded:
        enabled: false
      external:
        enabled: false
    etcd:
      embedded:
        enabled: false
  ingress:
    enabled: true
    host: api.vk8s4.gdo.priv
    annotations:
      ingress.kubernetes.io/backend-protocol: HTTPS
      ingress.kubernetes.io/ssl-passthrough: "true"
      ingress.kubernetes.io/ssl-redirect: "true"
    spec:
      ingressClassName: nginx-internal
    pathType: ImplementationSpecific
exportKubeConfig:
  context: "vk8s4"
  server: "https://api.vk8s4.gdo.priv"

k3s-state-store-connection secret

apiVersion: v1
data:
  K3S_DATASTORE_ENDPOINT: postgres://user:pass@host:5432/vk8s4
kind: Secret
metadata:
  name: k3s-state-store-connection
  namespace: vcluster-vk8s4
type: Opaque

[paalkr]

I guess this check needs to be refactored to allow for multiple replicas if using the k3s native kine implementation.
https://github.com/loft-sh/vcluster/blob/8494146a92f15bf35abb1d0995db6602cf6ed1c8/pkg/config/validation.go#L64C1-L66C3