Longer loading times for index headers with Thanos Receive compared to Thanos Sidecar

[roth-wine]

Thanos, Prometheus and Golang version used:

Thanos: v0.36.1
Prometheus: v2.53.1

Object Storage Provider:
Ceph Cluster

What happened:
We noticed that our Thanos Stores, which uses an Objectstorage as storage backend, has significant longer loading times per block/index when the are generated from the Thanos Receiver compared to Thanos Sidecar.
We then checked the two buckets, which both have a similar size and the blocks are also similar in size. For Tenant 1 we have 3112 index files with an average of 348MB. For tenant 2 we have 2993 index files with an average of 309MB per file. But this does not explain why Tenant1 takes on average 2-3 times as long to load per Index Header compared to Tenant2.

What you expected to happen:
Regardless of which data source the blocks are written from, according to the documentation they should make no difference to how the Thanos Store generates the index headers.
With similarly large buckets and a similarly large number of blocks to be loaded, almost the same speeds should be achieved when generating the index header/block.

How to reproduce it (as minimally and precisely as possible):
Compares a bucket which was filled with Thanos Receive data and one which was filled with Thanos Sidecar data. Then start a store for the receive bucket and a store for the sidecar bucket. You should now be able to recognise the differences.

Full logs to relevant components:

Receiver Blocks

Sample Output from thanos tools bucket inspect

|            ULID            |         FROM         |        UNTIL         |     RANGE     |   UNTIL-DOWN   |  #SERIES  |   #SAMPLES    |  #CHUNKS   | COMP-LEVEL | COMP-FAILED |                                                    LABELS                                                    | RESOLUTION |  SOURCE   |
|----------------------------|----------------------|----------------------|---------------|----------------|-----------|---------------|------------|------------|-------------|--------------------------------------------------------------------------------------------------------------|------------|-----------|
| 01J5ADJ0YDNACNJDHHZQ8569R9 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,390,745 | 6,632,884,192 | 53,817,971 | 4          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-1,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5AE1S6B4B3SQT2N2P90RFZE | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,470,793 | 6,822,144,336 | 54,662,046 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-20,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AENP6H3APP8HMAZ4YB1Q69 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,523,468 | 7,296,872,037 | 58,401,675 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-19,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AF6JVAACJHSJKDPD0PWT9S | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,473,601 | 6,940,360,938 | 55,575,188 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-22,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AFKCBDFXHGASHFXX0X53DN | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,472,729 | 6,954,088,819 | 55,707,100 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-16,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AG38NNJH1N6TYF2V1F9XE3 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,499,885 | 6,926,953,022 | 55,516,517 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-0,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5AGFCSBEHH2T4Z4W0GM2575 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,391,613 | 6,727,877,419 | 54,527,334 | 4          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-15,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AGWMQJX829DA1KFV7EQKSY | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,296,077 | 6,450,519,360 | 53,433,198 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-23,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AJ0C392CNMDJH9BEZFY3J4 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,483,324 | 6,940,353,514 | 55,577,142 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-6,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5AJW46CSWWCMF31R32P5JKN | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,520,486 | 7,222,286,006 | 57,850,518 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-5,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5AKJJ831QYYFX0JRCZHDER9 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,513,029 | 7,125,389,232 | 57,103,805 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-18,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AM79Y8R9R7GCKWWVZHEH99 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,447,667 | 6,800,683,513 | 54,452,452 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-13,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5ANEJ1KQ0NR41AG4PEZSFP7 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,507,697 | 7,077,556,168 | 56,662,247 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-8,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5APA35Y4C5R39ASY2T8GYGV | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,510,407 | 7,045,648,983 | 56,417,914 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-4,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5APY3B0XS4YQ16Q2EBWR9VK | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,426,950 | 6,718,865,003 | 53,820,753 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-7,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5AQF2AC5FRZM83AMB7FA4FB | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,479,027 | 6,856,283,653 | 54,953,937 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-10,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AQZME2KQ702WQZ9WHA9343 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,399,707 | 6,645,819,614 | 53,204,040 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-17,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5ARWPSASD1V3D59NCVC4CSM | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,550,251 | 7,261,090,323 | 58,164,387 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-14,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5ASDK72HZGNM4MBA4J5FSCG | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,515,668 | 7,131,415,720 | 57,106,606 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-2,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5AT1BGWF7355BJ1KAYDRKZ1 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,476,254 | 7,070,666,245 | 56,617,297 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-3,tenant_id=tenant1-storage  | 0s         | compactor |

Sidecar Blocks

Sample Output from thanos tools bucket inspect

|            ULID            |         FROM         |        UNTIL         |     RANGE     |  UNTIL-DOWN   |  #SERIES  |   #SAMPLES    |  #CHUNKS   | COMP-LEVEL | COMP-FAILED |                                                  LABELS                                                  | RESOLUTION |  SOURCE   |
|----------------------------|----------------------|----------------------|---------------|---------------|-----------|---------------|------------|------------|-------------|----------------------------------------------------------------------------------------------------------|------------|-----------|
| 01J5AAF5HPF2PFW5ABW85SF0WT | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s       | 1,167,658 | 5,209,825,975 | 46,609,953 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-10,prometheus_replica=prometheus-sidecar-cnodes-10-0,region=fsn1 | 0s         | compactor |
| 01J5AAXV5ATB0HMGVJR6W1FZ2V | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s       | 1,089,725 | 5,059,691,020 | 46,056,643 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-1,prometheus_replica=prometheus-sidecar-cnodes-1-0,region=fsn1   | 0s         | compactor |
| 01J5AB7V3H3J0E5FR1M8GA00GP | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | 192h0m0s      | 1,167,658 | 624,417,950   | 4,694,514  | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-10,prometheus_replica=prometheus-sidecar-cnodes-10-0,region=fsn1 | 5m0s       | compactor |
| 01J5ABQ09FP3JD4QK5F6C3JAX0 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | 192h0m0s      | 1,089,725 | 609,356,419   | 4,519,046  | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-1,prometheus_replica=prometheus-sidecar-cnodes-1-0,region=fsn1   | 5m0s       | compactor |
| 01J5AN61W5608MDYQDY4Q2NYWV | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.951s | -7h59m59.951s | 1,466,663 | 6,651,517,461 | 59,629,508 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-11,prometheus_replica=prometheus-sidecar-cnodes-11-0,region=fsn1 | 0s         | compactor |
| 01J5ANN2BX9XNR4M29YXWYBVDB | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.998s | -7h59m59.998s | 1,582,493 | 7,864,742,275 | 69,268,695 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-22,prometheus_replica=prometheus-sidecar-cnodes-22-0,region=fsn1 | 0s         | compactor |
| 01J5APAJ59PFEAZTAMACFPNV98 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.992s | -7h59m59.992s | 1,333,182 | 6,461,683,230 | 57,603,142 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-15,prometheus_replica=prometheus-sidecar-cnodes-15-0,region=fsn1 | 0s         | compactor |
| 01J5AQ1NPR8M6B4NGFMV12EHM3 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.998s | -7h59m59.998s | 1,613,122 | 7,945,622,683 | 70,102,189 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-19,prometheus_replica=prometheus-sidecar-cnodes-19-0,region=fsn1 | 0s         | compactor |
| 01J5AQQ0N1EXJ6X50G1YZSA4AE | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.978s | -7h59m59.978s | 1,330,199 | 6,189,685,122 | 55,563,851 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-12,prometheus_replica=prometheus-sidecar-cnodes-12-0,region=fsn1 | 0s         | compactor |
| 01J5AR5R0KP9FA2KDZP5SBDWVX | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.975s | -7h59m59.975s | 1,308,462 | 6,319,618,992 | 56,376,946 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-2,prometheus_replica=prometheus-sidecar-cnodes-2-0,region=fsn1   | 0s         | compactor |
| 01J5ARHMGKAK7JR90VGPDKGQDR | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.993s | -7h59m59.993s | 1,032,887 | 4,753,217,152 | 43,427,265 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-9,prometheus_replica=prometheus-sidecar-cnodes-9-0,region=fsn1   | 0s         | compactor |
| 01J5ARVN1S460E8G55ZC1VH0J5 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.952s | -7h59m59.952s | 879,900   | 4,269,481,051 | 37,960,012 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-24,prometheus_replica=prometheus-sidecar-cnodes-24-0,region=fsn1 | 0s         | compactor |
| 01J5ASA4NHPFQP41M52CN32SKH | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.993s | -7h59m59.993s | 1,423,181 | 7,007,498,147 | 62,191,611 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-14,prometheus_replica=prometheus-sidecar-cnodes-14-0,region=fsn1 | 0s         | compactor |
| 01J5ASTXDVACMQ8X28Z2WP07PE | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.999s | -7h59m59.999s | 1,319,988 | 6,271,030,141 | 56,402,900 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-3,prometheus_replica=prometheus-sidecar-cnodes-3-0,region=fsn1   | 0s         | compactor |
| 01J5ATBB4N72TSMSB6K3WA8HCM | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.991s | -7h59m59.991s | 1,306,804 | 6,213,659,211 | 55,874,472 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-5,prometheus_replica=prometheus-sidecar-cnodes-5-0,region=fsn1   | 0s         | compactor |
| 01J5ATSCJM8J7N0P2FC3H69QQV | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.998s | -7h59m59.998s | 1,430,156 | 6,383,481,921 | 57,123,055 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-17,prometheus_replica=prometheus-sidecar-cnodes-17-0,region=fsn1 | 0s         | compactor |
| 01J5AVAHJNP1AGX8CETWPDCNHM | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.992s | -7h59m59.992s | 1,230,953 | 5,984,898,253 | 53,706,705 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-13,prometheus_replica=prometheus-sidecar-cnodes-13-0,region=fsn1 | 0s         | compactor |
| 01J5AVPZ5XTCBP0091RV7ABH67 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.998s | -7h59m59.998s | 1,293,051 | 5,843,361,729 | 52,638,926 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-4,prometheus_replica=prometheus-sidecar-cnodes-4-0,region=fsn1   | 0s         | compactor |
| 01J5AW2NH655C3GZYTHDHPD84N | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.995s | -7h59m59.995s | 1,455,648 | 6,593,190,912 | 58,906,486 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-18,prometheus_replica=prometheus-sidecar-cnodes-18-0,region=fsn1 | 0s         | compactor |
| 01J5AWG07X66J4D30T82505G1V | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.985s | -7h59m59.985s | 1,073,459 | 4,956,575,766 | 45,101,988 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-8,prometheus_replica=prometheus-sidecar-cnodes-8-0,region=fsn1   | 0s         | compactor |

thanos_bucket_store_blocks_loaded

sum (thanos_bucket_store_blocks_loaded{region="fsn1", pod="store-test-0") by (namespace, region)

{namespace="monitoring-tenant1-storage", region="fsn1"} 2497
{namespace="scrape-tenant2-cnodes", region="fsn1"} 2491

thanos_bucket_store_indexheader_load_duration_seconds_sum

Green: Tenant1
Blue: Tenant2

thanos_objstore_bucket_operation_transferred_bytes_sum

Green: Tenant1
Blue: Tenant2

[yeya24]

Is this always reproducible or you discovered this issue the first time?

For Tenant 1 we have 3112 index files with an average of 348MB

Have you checked the largest index file size?

[roth-wine]

Hey, yeah this is always reproducible even with new buckets. Currently we scrape and receive the same data for one tenant and also for this tenant this behavior is reproducible.

The largest index file is about 800MB.