Cannot create metrics-generator because of read-only filesystem

[Jeansen]

I have the following part in my a monolith operator yaml:

...
  extraConfig:                           
    tempo:
      metrics_generator:
        processor:
          span_metrics:
            enable_target_info: true
          local_blocks:
            filter_server_spans: false
            flush_to_storage: true
        storage:
          path: "/tmp"
          remote_write:
            - url: http://grafana-mimir-gateway.default.svc/api/v1/push
              send_exemplars: true
...

In the logs, I see that everything comes up fine, except:

level=warn ts=2025-02-14T23:42:15.753098235Z caller=server.go:1187 method=/tempopb.MetricsGenerator/PushSpans duration=211.297µs msg=gRPC err="could not create directory for metrics WAL: mkdir /tmp/single-tenant: read-only file system"

The operator gives me no way to add additional volumes or overwrite the read-only root file system setting. So, currently there is no way to activate the metrics_generator because it needs a path for the WAL to write to.

[andreasgerstmayr]

Can you update your config to use /var/tempo/generator/wal as the path for the WAL? /var/tempo is mounted as a tmpfs for in-memory-mode, and as a PV otherwise.

In the past, I used the following spec successfully:

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoMonolithic
metadata:
  name: sample
spec:
  extraConfig:
    tempo:
      metrics_generator:
        storage:
          path: /var/tempo/generator/wal
          remote_write:
          - url: http://kube-prometheus-stack1-prometheus.kube-prometheus-stack1:9090/api/v1/write
            send_exemplars: true
      overrides:
        defaults:
          metrics_generator:
            processors: [service-graphs, span-metrics]