Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[query-frontend] Unable to connect to Redis(elasticache) 6.2.6 #7672

Open
beramod opened this issue Aug 28, 2024 · 1 comment
Open

[query-frontend] Unable to connect to Redis(elasticache) 6.2.6 #7672

beramod opened this issue Aug 28, 2024 · 1 comment

Comments

@beramod
Copy link

beramod commented Aug 28, 2024

Thanos, Prometheus and Golang version used: Thanos 0.34.1, AWS Elasticache 6.2.6

Object Storage Provider: AWS

What happened: I'm trying to configure cache (Redis) for Query-Frontend. but I'm encountering an "error: creating redis client: context deadline exceeded", and the connection fails. I've configured the same settings for the index and bucket cache, and the cache connection for the store component is working fine.

What you expected to happen: Query-Frontend can connect to Redis

How to reproduce it (as minimally and precisely as possible):

  • Deploy query frontend version 0.34.1 with the following cache config:
type: REDIS
config:
  addr: "url:6379"
  username: ""
  password: "$(PASSWORD)"
  db: 0
  dial_timeout: 10s
  read_timeout: 3s
  write_timeout: 3s
  max_get_multi_concurrency: 100
  get_multi_batch_size: 100
  max_set_multi_concurrency: 100
  set_multi_batch_size: 100
  tls_enabled: true
  tls_config:
    insecure_skip_verify: true
  cache_size: 0
  max_async_buffer_size: 10000
  max_async_concurrency: 20
  expiration: 1h

Full logs to relevant components:

Logs

{"caller":"main.go:67","level":"debug","msg":"maxprocs: Updating GOMAXPROCS=[1]: using minimum allowed GOMAXPROCS","ts":"2024-08-28T09:31:42.966494789Z"}
{"caller":"main.go:135","err":"creating redis client: context deadline exceeded\ngithub.com/thanos-io/thanos/internal/cortex/chunk/cache.New\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/internal/cortex/chunk/cache/cache.go:108\ngithub.com/thanos-io/thanos/internal/cortex/querier/queryrange.NewResultsCacheMiddleware\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/internal/cortex/querier/queryrange/results_cache.go:187\ngithub.com/thanos-io/thanos/pkg/queryfrontend.newQueryRangeTripperware\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/pkg/queryfrontend/roundtrip.go:208\ngithub.com/thanos-io/thanos/pkg/queryfrontend.NewTripperware\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/pkg/queryfrontend/roundtrip.go:60\nmain.runQueryFrontend\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/query_frontend.go:299\nmain.registerQueryFrontend.func1\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/query_frontend.go:172\nmain.main\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/main.go:133\nruntime.main\n\t/opt/bitnami/go/src/runtime/proc.go:267\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650\ncreate results cache middleware\ngithub.com/thanos-io/thanos/pkg/queryfrontend.newQueryRangeTripperware\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/pkg/queryfrontend/roundtrip.go:220\ngithub.com/thanos-io/thanos/pkg/queryfrontend.NewTripperware\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/pkg/queryfrontend/roundtrip.go:60\nmain.runQueryFrontend\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/query_frontend.go:299\nmain.registerQueryFrontend.func1\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/query_frontend.go:172\nmain.main\n\
t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/main.go:133\nruntime.main\n\t/opt/bitnami/go/src/runtime/proc.go:267\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650\nsetup tripperwares\nmain.runQueryFrontend\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/query_frontend.go:301\nmain.registerQueryFrontend.func1\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/query_frontend.go:172\nmain.main\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/main.go:133\nruntime.main\n\t/opt/bitnami/go/src/runtime/proc.go:267\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650\npreparing query-frontend command failed\nmain.main\n\t/bitnami/blacksmith-sandox/thanos-0.34.1/src/github.com/thanos-io/thanos/cmd/thanos/main.go:135\nruntime.main\n\t/opt/bitnami/go/src/runtime/proc.go:267\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650","level":"error","ts":"2024-08-28T09:31:45.973964778Z"}

Anything else we need to know:
I found a similar issue in the link below(#6094)

didukh86 added a commit to didukh86/thanos that referenced this issue Aug 28, 2024
didukh86 added a commit to didukh86/thanos that referenced this issue Aug 28, 2024
didukh86 added a commit to didukh86/thanos that referenced this issue Aug 28, 2024
Issue: thanos-io#7672

Signed-off-by: didukh86 <[email protected]>
Signed-off-by: didukh86 <[email protected]>
Signed-off-by: didukh86 <[email protected]>
didukh86 added a commit to didukh86/thanos that referenced this issue Sep 4, 2024
Issue: thanos-io#7672

Signed-off-by: didukh86 <[email protected]>
Signed-off-by: didukh86 <[email protected]>
Signed-off-by: didukh86 <[email protected]>
Signed-off-by: Taras Didukh <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants
@beramod and others