Basic Auth Bootstrap Not Working In 0.9.0

[ibraheemalayan]

Hi team,

I have a very simple setup and it can be easily replicated:

started by installing the operator on a fresh k8s cluster using a helm chart:

kubectl create -f https://solr.apache.org/operator/downloads/crds/v0.9.0/all-with-dependencies.yaml
helm install solr-operator apache-solr/solr-operator --version 0.9.0

then kubectl apply of the following SolrCloud config

# specs: https://apache.github.io/solr-operator/docs/solr-cloud/solr-cloud-crd.html
apiVersion: solr.apache.org/v1beta1
kind: SolrCloud
metadata:
  name: search-cluster
spec:
  # change the date below to trigger a manual restart
  customSolrKubeOptions:
    podOptions:
      annotations:
        manualrestart: "2025-02-06T07:57:00Z"
  replicas: 2
  solrJavaMem: -Xms512M -Xmx1G
  solrImage:
    tag: 9.8.0
  solrSecurity:
    authenticationType: Basic
  dataStorage:
    persistent:
      reclaimPolicy: Delete
      pvcTemplate:
        spec:
          resources:
            requests:
              storage: 6Gi
  solrAddressability:
    podPort: 8983
    commonServicePort: 8983

as you can see, basic auth is supposed to be enabled, yet the cluster has no authentication.

I tried deleting the SolrCloud and recreating it multiple times ( also deleted all PVCs ), the security.json is always empty
The bootstrape security.json secret is created but it seems like its never used.

Here some debugging commands and their outputs, might help understand the issue:

kubectl describe solrcloud/search-cluster

Name:         search-cluster
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  solr.apache.org/v1beta1
Kind:         SolrCloud
Metadata:
  Creation Timestamp:  2025-02-06T10:48:25Z
  Finalizers:
    storage.finalizers.solr.apache.org
  Generation:        2
  Resource Version:  38419295
  UID:               3f94f03a-9d8c-4988-a991-a4fa225b0b8e
Spec:
  Availability:
    Pod Disruption Budget:
      Enabled:  true
      Method:   ClusterWide
  Busy Box Image:
    Repository:  library/busybox
    Tag:         1.28.0-glibc
  Custom Solr Kube Options:
    Pod Options:
      Annotations:
        Manualrestart:  2025-02-06T07:57:00Z
      Default Init Container Resources:
      Resources:
  Data Storage:
    Persistent:
      Pvc Template:
        Metadata:
        Spec:
          Resources:
            Requests:
              Storage:  6Gi
      Reclaim Policy:   Delete
  Replicas:             2
  Scaling:
    Populate Pods On Scale Up:  true
    Vacate Pods On Scale Down:  true
  Solr Addressability:
    Common Service Port:  8983
    Pod Port:             8983
  Solr Image:
    Repository:    library/solr
    Tag:           9.8.0
  Solr Java Mem:   -Xms512M -Xmx1G
  Solr Log Level:  INFO
  Solr Security:
    Authentication Type:  Basic
  Update Strategy:
    Managed:
    Method:  Managed
  Zookeeper Ref:
    Provided:
      Admin Server Service:
      Chroot:  /
      Client Service:
      Config:
      Headless Service:
      Image:
        Pull Policy:             IfNotPresent
        Repository:              pravega/zookeeper
      Max Unavailable Replicas:  1
      Replicas:                  3
      Zookeeper Pod Policy:
        Resources:
Status:
  Internal Common Address:  http://search-cluster-solrcloud-common.default:8983
  Pod Selector:             solr-cloud=search-cluster,technology=solr-cloud
  Ready Replicas:           2
  Replicas:                 2
  Solr Nodes:
    Internal Address:        http://search-cluster-solrcloud-0.search-cluster-solrcloud-headless.default:8983
    Name:                    search-cluster-solrcloud-0
    Node Name:               gke-main-cluster-main-pool-0d3ce34b-nrmn
    Ready:                   true
    Scheduled For Deletion:  false
    Spec Up To Date:         true
    Version:                 9.8.0
    Internal Address:        http://search-cluster-solrcloud-1.search-cluster-solrcloud-headless.default:8983
    Name:                    search-cluster-solrcloud-1
    Node Name:               gke-main-cluster-main-pool-0d3ce34b-nrmn
    Ready:                   true
    Scheduled For Deletion:  false
    Spec Up To Date:         true
    Version:                 9.8.0
  Up To Date Nodes:          2
  Version:                   9.8.0
  Zookeeper Connection Info:
    Chroot:                      /
    External Connection String:  N/A
    Internal Connection String:  search-cluster-solrcloud-zookeeper-0.search-cluster-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,search-cluster-solrcloud-zookeeper-1.search-cluster-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,search-cluster-solrcloud-zookeeper-2.search-cluster-solrcloud-zookeeper-headless.default.svc.cluster.local:2181
Events:                          <none>

kubectl get pods | grep solr

search-cluster-solrcloud-0                          1/1     Running   0          8m
search-cluster-solrcloud-1                          1/1     Running   0          15m
search-cluster-solrcloud-zookeeper-0                1/1     Running   0          15m
search-cluster-solrcloud-zookeeper-1                1/1     Running   0          14m
search-cluster-solrcloud-zookeeper-2                1/1     Running   0          14m
solr-operator-65bd76858c-dj54z                      1/1     Running   0          4d12h
solr-operator-zookeeper-operator-7cf584d8b6-5vlzr   1/1     Running   0          4d12h

kubectl get secrets

search-cluster-solrcloud-basic-auth           kubernetes.io/basic-auth   2      16m
search-cluster-solrcloud-security-bootstrap   Opaque                     3      16m
sh.helm.release.v1.solr-operator.v1           helm.sh/release.v1         1      7d23h

kubectl get secrets/search-cluster-solrcloud-basic-auth -o yaml

apiVersion: v1
data:
  password: amkodUctamRWYTJIM3cpRg==
  username: azhzLW9wZXI=
kind: Secret
metadata:
  creationTimestamp: "2025-02-06T10:09:01Z"
  labels:
    solr-cloud: search-cluster
  name: search-cluster-solrcloud-basic-auth
  namespace: default
  ownerReferences:
  - apiVersion: solr.apache.org/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: SolrCloud
    name: search-cluster
    uid: 0f94b8ff-134f-459c-903f-cc9a8312992e
  resourceVersion: "38390607"
  uid: feeb1481-474c-48bb-bbd4-4c3805e6a9bf
type: kubernetes.io/basic-auth

kubectl get secrets/search-cluster-solrcloud-security-bootstrap -o yaml

apiVersion: v1
data:
  admin: clUlcyVGQVtTQiN1aEdAUw==
  security.json: ewogICAgICAiYXV0aGVudGljYXRpb24iOnsKICAgICAgICAiYmxvY2tVbmtub3duIjogZmFsc2UsCiAgICAgICAgImNsYXNzIjoic29sci5CYXNpY0F1dGhQbHVnaW4iLAogICAgICAgICJjcmVkZW50aWFscyI6IHsiYWRtaW4iOiJDYldWUVV6L1ZjTFdpczQreUpCUEpYMlg5OXg4YWYyMUhuV1hVOGJEc0hBPSAxTWhMaEJtRzZWUmx2M3NPOHZMZHFybjV6d3UxTXRKOXVMWmhUMi84S1Q0PSIsIms4cy1vcGVyIjoiNkxBLzdNbHk1T0NTaVpOeWFheHhJQ1RTa3FrbGZkWXpGSW40QXBONFlVaz0gWndBT3RYaENtYnhZenhNbHB5VGR1dU5jbjZWTC9DK0ZtREZ4NkhseVRvVT0iLCJzb2xyIjoiRE9IMmxUdnh5aEFtdFZpVEloVDYvT2hkd3E4eVQzOFhNNUhPckp3eHNrRT0gWDBHakVUUXY5eGhmYVF1dGRyQkRhU3dBSVdMd0VFd2hLRXV0YXJxdWd0UT0ifSwKICAgICAgICAicmVhbG0iOiJTb2xyIEJhc2ljIEF1dGgiLAogICAgICAgICJmb3J3YXJkQ3JlZGVudGlhbHMiOiBmYWxzZQogICAgICB9LAogICAgICAiYXV0aG9yaXphdGlvbiI6IHsKICAgICAgICAiY2xhc3MiOiAic29sci5SdWxlQmFzZWRBdXRob3JpemF0aW9uUGx1Z2luIiwKICAgICAgICAidXNlci1yb2xlIjogewogICAgICAgICAgImFkbWluIjogWyJhZG1pbiIsICJrOHMiXSwKICAgICAgICAgICJrOHMtb3BlciI6IFsiazhzIl0sCiAgICAgICAgICAic29sciI6IFsidXNlcnMiLCAiazhzIl0KICAgICAgICB9LAogICAgICAgICJwZXJtaXNzaW9ucyI6IFsKICAgICAgICAgIHsgIm5hbWUiOiAiazhzLXByb2JlLTAiLCAicm9sZSI6bnVsbCwgImNvbGxlY3Rpb24iOiBudWxsLCAicGF0aCI6Ii9hZG1pbi9pbmZvL3N5c3RlbSIgfSwgeyAibmFtZSI6ICJrOHMtcHJvYmUtMSIsICJyb2xlIjpudWxsLCAiY29sbGVjdGlvbiI6IG51bGwsICJwYXRoIjoiL2FkbWluL2luZm8vaGVhbHRoIiB9LAogICAgICAgICAgeyAibmFtZSI6ICJrOHMtc3RhdHVzIiwgInJvbGUiOiJrOHMiLCAiY29sbGVjdGlvbiI6IG51bGwsICJwYXRoIjoiL2FkbWluL2NvbGxlY3Rpb25zIiB9LAogICAgICAgICAgeyAibmFtZSI6ICJrOHMtbWV0cmljcyIsICJyb2xlIjoiazhzIiwgImNvbGxlY3Rpb24iOiBudWxsLCAicGF0aCI6Ii9hZG1pbi9tZXRyaWNzIiB9LAogICAgICAgICAgeyAibmFtZSI6ICJrOHMtemsiLCAicm9sZSI6Ims4cyIsICJjb2xsZWN0aW9uIjogbnVsbCwgInBhdGgiOiIvYWRtaW4vem9va2VlcGVyL3N0YXR1cyIgfSwKICAgICAgICAgIHsgIm5hbWUiOiAiazhzLXBpbmciLCAicm9sZSI6Ims4cyIsICJjb2xsZWN0aW9uIjogIioiLCAicGF0aCI6Ii9hZG1pbi9waW5nIiB9LAogICAgICAgICAgeyAibmFtZSI6ICJyZWFkIiwgInJvbGUiOlsiYWRtaW4iLCJ1c2VycyJdIH0sCiAgICAgICAgICB7ICJuYW1lIjogInVwZGF0ZSIsICJyb2xlIjpbImFkbWluIl0gfSwKICAgICAgICAgIHsgIm5hbWUiOiAic2VjdXJpdHktcmVhZCIsICJyb2xlIjogWyJhZG1pbiJdIH0sCiAgICAgICAgICB7ICJuYW1lIjogInNlY3VyaXR5LWVkaXQiLCAicm9sZSI6IFsiYWRtaW4iXSB9LAogICAgICAgICAgeyAibmFtZSI6ICJhbGwiLCAicm9sZSI6WyJhZG1pbiJdIH0KICAgICAgICBdCiAgICAgIH0KICAgIH0=
  solr: cV1QeVVsdkRHNkFySHAzSw==
kind: Secret
metadata:
  creationTimestamp: "2025-02-06T10:09:01Z"
  labels:
    solr-cloud: search-cluster
  name: search-cluster-solrcloud-security-bootstrap
  namespace: default
  ownerReferences:
  - apiVersion: solr.apache.org/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: SolrCloud
    name: search-cluster
    uid: 0f94b8ff-134f-459c-903f-cc9a8312992e
  resourceVersion: "38390610"
  uid: cbdb9c8e-dde5-4c09-a9c2-157e6a4727ea
type: Opaque

[ibraheemalayan]

⭕️ UPDATE:
I deleted the SolrCloud, recreated it with Solr 8.11.4 and it worked, then deleted that, and created with 9.6.1 and it also worked.
So I deleted it again (without deleting the zookeeper PVCs) and recreated it with 9.8.0 and it worked.

But clearly there is an issue with 9.8.0

[Calcagiara]

Hey there, having the same problem here. I will add that version 9.7.0 works fine with a fresh new deploy too, so I'm guessing it's an issue related specifically with the 9.8.0 version.

[idjemaoune]

I encountered the same issue with version 9.7.0 when creating my cluster from scratch with this version. However, upgrading from version 9.6.1 (with authentication already in place) to later versions works correctly.