[helm-chart] Enabling bloom increases loki-backend's ephemeral volume usage

[kimxogus]

Describe the bug
I'm using loki with SimpleScalable mode in helm chart.
I recently enabled bloom, and loki-backend's ephemeral volume usage is over 30Gb, and I found that bloom block files in /tmp/ directory is using that 30gb.

To Reproduce
Steps to reproduce the behavior:

1. Deploy SimpleScalable loki with bloom enabled in helm chart 6.18.0

   limits_config:
       bloom_creation_enabled: true
       bloom_gateway_enable_filtering: true
       bloom_split_series_keyspace_by: 1024
   
   bloom_build:
     enabled: true
   
   bloom_gateway:
     enabled: true
     worker_concurrency: 4
     block_query_concurrency: 8

2. Your loki-backend's /tmp/ volume usage grows as your log indexing and query usage.

Expected behavior
loki-backend's ephemeral volume usage is minimized

• bloom blocks are written in directory other than /tmp/ for PVC
• bloom blocks are written in other than ephemeral volume(PVC or object storage)

Environment:

• Infrastructure: AWS EKS 1.30.
• Deployment tool: loki Helm Chart 6.18.0

Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.

$ ls -alh /tmp/bloom-block-{12345678}
total 32G
drwx--S---    2 loki     loki        4.0K Oct 30 11:53 .
drwxrwsrwx    4 root     loki        4.0K Oct 30 23:59 ..
-rw-r--r--    1 loki     loki       31.6G Oct 30 23:29 bloom
-rw-r--r--    1 loki     loki      140.4K Oct 30 23:29 series