I would like to know why the Thanos server failed to collect data.

[SungYil]

we running two k8s clusters.

Thanos sends metrics from cluster 1 to cluster 2.
We installed Thanos using helm.

cluster 2 version : prometheus 56.3.0, thanos 12.23.0

cluster 1 :

prometheus:
  thanosService:
    enabled: true
  extraSecret:
    name: bucket-config
    data:
      objstore.yml: |
        type: s3
        config:
          bucket: aws-s3
          region: ap-northeast-2
          endpoint: s3.ap-northeast-2.amazonaws.com
          access_key: $(AWS_ACCESS_KEY)
          secret_key: $(AWS_SECRET_KEY)
          insecure: false
          signature_version2: false
  prometheusSpec:
    containers:
      - name: thanos-sidecar
        env:
          - name: AWS_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                key: AWS_ACCESS_KEY
                name: aws-secret
          - name: AWS_SECRET_KEY
            valueFrom:
              secretKeyRef:
                key: AWS_SECRET_KEY
                name: aws-secret
    externalLabels:
      cluster: "cluster1"
    disableCompaction: true
    thanos:
      objectStorageConfig:
        existingSecret:
          name: bucket-config
          key: objstore.yml
grafana:
  enabled: false
  adminPassword: pass
  defaultDashboardsEnabled: false
  sidecar:
    datasources:
      enabled: true
      defaultDatasourceEnabled: false
defaultRules:
  create: false
alertmanager:
  enabled: false
prometheusOperator:
  thanosImage:
    tag: v0.24.0
kubeApiServer:
  serviceMonitor:
    metricRelabelings:
      # Drop excessively noisy apiserver buckets.
      - sourceLabels:
          - __name__
        action: drop
        regex: 'etcd_.*'
      - sourceLabels:
          - __name__
        action: drop
        regex: 'apiserver_.*'

cluster 2:

image:
  tag: 0.24.0
objstoreConfig: |-
  type: s3
  config:
    bucket: aws-s3
    region: ap-northeast-2
    endpoint: s3.ap-northeast-2.amazonaws.com
    access_key: $(AWS_ACCESS_KEY)
    secret_key: $(AWS_SECRET_KEY)
    insecure: false
    signature_version2: false
querier:
  stores:
    - cluster1-domain:80
  service:
    type: ClusterIP
    ports:
      http: 9090
    nodePorts:
      http: "31254"
    cplusterIP: ""
bucketweb:
  enabled: false
compactor:
  enabled: false
storegateway:
  enabled: true
  extraEnvVars:
    - name: AWS_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          key: AWS_ACCESS_KEY
          name: aws-secret
    - name: AWS_SECRET_KEY
      valueFrom:
        secretKeyRef:
          key: AWS_SECRET_KEY
          name: aws-secret
ruler:
  enabled: false

cluster 2 thanos-query pod logs :

• query: error DeadlineExceeded missing error details
• receive series from addr: labelsets: {cluster="cluster 1", prometheus="monitoring/prometheus-kube-prometheus-prometheus", prometheus_replica="prometheus-prometheus-kube-prometheus-prometheus-0"} mint: 0 maxt: 9223372036854775807: rpc error: code = canceled desc = grpc: the client connection is closing" msg="returning partial response"
• removing endpoint because it's unhealthy or does not exist
• no storeapis matched for this query

Today I suddenly started not being able to collect data.

I changed it to the content below and it was resolved.

cluster 1 yaml:

    thanos:
      objectStorageConfig:
        name: bucket-config
        key: objstore.yml


->


   thanos:
      objectStorageConfig:
        existingSecret:
          name: bucket-config
          key: objstore.yml

The helm version was not separately updated.
I want to know why it wasn't collected.