Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix[immich]: tpl not respecting container resources #32023

Open
3 tasks done
WhiffyMuffinz opened this issue Feb 13, 2025 · 1 comment
Open
3 tasks done

fix[immich]: tpl not respecting container resources #32023

WhiffyMuffinz opened this issue Feb 13, 2025 · 1 comment
Assignees
@stavros-k
Labels
bug Something isn't working

Comments

@WhiffyMuffinz
Copy link

Chart Name

immich

Operating System

Talos-OS 1.7

Deployment Method

FluxCD

Chart Version

17.48.1

Kubernetes Events

LAST SEEN             TYPE      REASON                   OBJECT                                         MESSAGE
24m                   Normal    UpdateCertificate        Ingress/immich                                 Successfully updated Certificate "immich-tls-0"
24m                   Normal    SuccessfulDelete         ReplicaSet/immich-74ffcfc46c                   Deleted pod: immich-74ffcfc46c-p2522
24m                   Normal    Killing                  Pod/immich-microservices-d6d6bf4c5-7t9tr       Stopping container immich
24m                   Normal    SuccessfulDelete         ReplicaSet/immich-microservices-d6d6bf4c5      Deleted pod: immich-microservices-d6d6bf4c5-7t9tr
24m                   Normal    ScalingReplicaSet        Deployment/immich-machinelearning              Scaled down replica set immich-machinelearning-75f559666b to 0 from 1
24m                   Normal    ScalingReplicaSet        Deployment/immich-microservices                Scaled down replica set immich-microservices-d6d6bf4c5 to 0 from 1
24m                   Normal    Killing                  Pod/immich-redis-0                             Stopping container immich-redis
24m                   Normal    SuccessfulDelete         ReplicaSet/immich-machinelearning-75f559666b   Deleted pod: immich-machinelearning-75f559666b-q2wrb
24m                   Normal    Killing                  Pod/immich-machinelearning-75f559666b-q2wrb    Stopping container immich
24m (x6 over 2d22h)   Normal    SuccessfulDelete         StatefulSet/immich-redis                       delete Pod immich-redis-0 in StatefulSet immich-redis successful
24m                   Normal    Killing                  Pod/immich-74ffcfc46c-p2522                    Stopping container immich
24m                   Normal    ScalingReplicaSet        Deployment/immich                              Scaled down replica set immich-74ffcfc46c to 0 from 1
24m                   Warning   FailedAttachVolume       Pod/immich-microservices-5f7598bf4d-tz68g      Multi-Attach error for volume "pvc-f8c08f4c-e7a4-44cc-997f-053e29a48180" Volume is already exclusively attached to one node and can't be attached to another
24m                   Warning   FailedAttachVolume       Pod/immich-microservices-5f7598bf4d-tz68g      Multi-Attach error for volume "pvc-d66e8a77-3707-45d2-819d-e99d20b88af3" Volume is already exclusively attached to one node and can't be attached to another
24m                   Warning   FailedAttachVolume       Pod/immich-77f98ff5bd-4p5bv                    Multi-Attach error for volume "pvc-f8c08f4c-e7a4-44cc-997f-053e29a48180" Volume is already exclusively attached to one node and can't be attached to another
24m                   Normal    ScalingReplicaSet        Deployment/immich                              Scaled up replica set immich-77f98ff5bd to 1
24m                   Warning   FailedAttachVolume       Pod/immich-microservices-5f7598bf4d-tz68g      Multi-Attach error for volume "pvc-6e14e326-5342-4431-a5e4-88c085b0becb" Volume is already exclusively attached to one node and can't be attached to another
24m                   Warning   FailedAttachVolume       Pod/immich-microservices-5f7598bf4d-tz68g      Multi-Attach error for volume "pvc-5c1e0400-0213-4c60-88f5-e1230d8bcb31" Volume is already exclusively attached to one node and can't be attached to another
24m                   Warning   FailedAttachVolume       Pod/immich-microservices-5f7598bf4d-tz68g      Multi-Attach error for volume "pvc-c5098569-4474-4937-b6a8-267071bf568a" Volume is already exclusively attached to one node and can't be attached to another
24m                   Warning   FailedAttachVolume       Pod/immich-microservices-5f7598bf4d-tz68g      Multi-Attach error for volume "pvc-9dd91932-1ead-4b7f-b057-71a3d3a685e2" Volume is already exclusively attached to one node and can't be attached to another
24m (x8 over 3d3h)    Normal    SuccessfulCreate         StatefulSet/immich-redis                       create Pod immich-redis-0 in StatefulSet immich-redis successful
24m                   Normal    SuccessfulCreate         ReplicaSet/immich-microservices-5f7598bf4d     Created pod: immich-microservices-5f7598bf4d-tz68g
24m                   Normal    ScalingReplicaSet        Deployment/immich-machinelearning              Scaled up replica set immich-machinelearning-d45ff9cd7 to 1
24m                   Warning   FailedAttachVolume       Pod/immich-77f98ff5bd-4p5bv                    Multi-Attach error for volume "pvc-9dd91932-1ead-4b7f-b057-71a3d3a685e2" Volume is already exclusively attached to one node and can't be attached to another
24m                   Normal    SuccessfulCreate         ReplicaSet/immich-machinelearning-d45ff9cd7    Created pod: immich-machinelearning-d45ff9cd7-72sgv
24m                   Normal    SuccessfulCreate         ReplicaSet/immich-77f98ff5bd                   Created pod: immich-77f98ff5bd-4p5bv
24m                   Warning   FailedAttachVolume       Pod/immich-77f98ff5bd-4p5bv                    Multi-Attach error for volume "pvc-6e14e326-5342-4431-a5e4-88c085b0becb" Volume is already exclusively attached to one node and can't be attached to another
24m                   Warning   FailedAttachVolume       Pod/immich-77f98ff5bd-4p5bv                    Multi-Attach error for volume "pvc-5c1e0400-0213-4c60-88f5-e1230d8bcb31" Volume is already exclusively attached to one node and can't be attached to another
24m                   Warning   FailedAttachVolume       Pod/immich-77f98ff5bd-4p5bv                    Multi-Attach error for volume "pvc-c5098569-4474-4937-b6a8-267071bf568a" Volume is already exclusively attached to one node and can't be attached to another
24m                   Normal    ScalingReplicaSet        Deployment/immich-microservices                Scaled up replica set immich-microservices-5f7598bf4d to 1
24m                   Warning   FailedAttachVolume       Pod/immich-77f98ff5bd-4p5bv                    Multi-Attach error for volume "pvc-d66e8a77-3707-45d2-819d-e99d20b88af3" Volume is already exclusively attached to one node and can't be attached to another
24m                   Normal    Scheduled                Pod/immich-machinelearning-d45ff9cd7-72sgv     Successfully assigned immich/immich-machinelearning-d45ff9cd7-72sgv to k8s-worker-01
24m                   Normal    Scheduled                Pod/immich-77f98ff5bd-4p5bv                    Successfully assigned immich/immich-77f98ff5bd-4p5bv to k8s-worker-01
24m                   Normal    Scheduled                Pod/immich-microservices-5f7598bf4d-tz68g      Successfully assigned immich/immich-microservices-5f7598bf4d-tz68g to k8s-worker-01
24m                   Normal    Scheduled                Pod/immich-redis-0                             Successfully assigned immich/immich-redis-0 to k8s-worker-01
24m                   Normal    Created                  Pod/immich-machinelearning-d45ff9cd7-72sgv     Created container immich-init-wait-server
24m                   Normal    Pulled                   Pod/immich-machinelearning-d45ff9cd7-72sgv     Container image "tccr.io/tccr/alpine:v3.21.2@sha256:b25393956a2e693711e9de978f5a884c88f51064d64bb48e7f699fbe75d430b6" already present on machine
24m                   Normal    Pulling                  Pod/immich-redis-0                             Pulling image "public.ecr.aws/bitnami/redis:7.4.2@sha256:46244c9a40a5c5cce081bbd1fad1c72c484708d6aba3211629c65b6404cef156"
24m                   Normal    Started                  Pod/immich-machinelearning-d45ff9cd7-72sgv     Started container immich-init-wait-server
24m                   Normal    Pulled                   Pod/immich-redis-0                             Successfully pulled image "public.ecr.aws/bitnami/redis:7.4.2@sha256:46244c9a40a5c5cce081bbd1fad1c72c484708d6aba3211629c65b6404cef156" in 7.371s (7.371s including waiting). Image size: 50936732 bytes.
24m                   Normal    Created                  Pod/immich-redis-0                             Created container immich-redis
24m                   Normal    Started                  Pod/immich-redis-0                             Started container immich-redis
24m                   Normal    SuccessfulAttachVolume   Pod/immich-77f98ff5bd-4p5bv                    AttachVolume.Attach succeeded for volume "pvc-d66e8a77-3707-45d2-819d-e99d20b88af3"
24m                   Normal    SuccessfulAttachVolume   Pod/immich-microservices-5f7598bf4d-tz68g      AttachVolume.Attach succeeded for volume "pvc-d66e8a77-3707-45d2-819d-e99d20b88af3"
24m                   Normal    SuccessfulAttachVolume   Pod/immich-microservices-5f7598bf4d-tz68g      AttachVolume.Attach succeeded for volume "pvc-c5098569-4474-4937-b6a8-267071bf568a"
24m                   Normal    SuccessfulAttachVolume   Pod/immich-77f98ff5bd-4p5bv                    AttachVolume.Attach succeeded for volume "pvc-c5098569-4474-4937-b6a8-267071bf568a"
24m                   Normal    SuccessfulAttachVolume   Pod/immich-77f98ff5bd-4p5bv                    AttachVolume.Attach succeeded for volume "pvc-9dd91932-1ead-4b7f-b057-71a3d3a685e2"
24m                   Normal    SuccessfulAttachVolume   Pod/immich-77f98ff5bd-4p5bv                    AttachVolume.Attach succeeded for volume "pvc-5c1e0400-0213-4c60-88f5-e1230d8bcb31"
24m                   Normal    SuccessfulAttachVolume   Pod/immich-77f98ff5bd-4p5bv                    AttachVolume.Attach succeeded for volume "pvc-6e14e326-5342-4431-a5e4-88c085b0becb"
24m                   Normal    SuccessfulAttachVolume   Pod/immich-microservices-5f7598bf4d-tz68g      AttachVolume.Attach succeeded for volume "pvc-5c1e0400-0213-4c60-88f5-e1230d8bcb31"
24m                   Normal    SuccessfulAttachVolume   Pod/immich-microservices-5f7598bf4d-tz68g      AttachVolume.Attach succeeded for volume "pvc-6e14e326-5342-4431-a5e4-88c085b0becb"
24m                   Normal    SuccessfulAttachVolume   Pod/immich-microservices-5f7598bf4d-tz68g      AttachVolume.Attach succeeded for volume "pvc-9dd91932-1ead-4b7f-b057-71a3d3a685e2"
24m                   Normal    SuccessfulAttachVolume   Pod/immich-microservices-5f7598bf4d-tz68g      AttachVolume.Attach succeeded for volume "pvc-f8c08f4c-e7a4-44cc-997f-053e29a48180"
24m                   Normal    SuccessfulAttachVolume   Pod/immich-77f98ff5bd-4p5bv                    AttachVolume.Attach succeeded for volume "pvc-f8c08f4c-e7a4-44cc-997f-053e29a48180"
24m                   Normal    Pulled                   Pod/immich-77f98ff5bd-4p5bv                    Container image "tccr.io/tccr/db-wait-redis:v1.1.0@sha256:11cbaee0ce92bc243071f31f4fb8bc30c2735ee34ead52cd842713c77bdf8739" already present on machine
24m                   Normal    Created                  Pod/immich-77f98ff5bd-4p5bv                    Created container immich-system-redis-wait
24m                   Normal    Pulled                   Pod/immich-77f98ff5bd-4p5bv                    Container image "tccr.io/tccr/db-wait-postgres:v1.1.0@sha256:5e95119261f8184663b618bf53db254fad31a369d74169155406d0472cb7680c" already present on machine
24m                   Normal    Created                  Pod/immich-77f98ff5bd-4p5bv                    Created container immich-system-cnpg-wait
24m                   Normal    Started                  Pod/immich-microservices-5f7598bf4d-tz68g      Started container immich-init-wait-server
24m                   Normal    Created                  Pod/immich-microservices-5f7598bf4d-tz68g      Created container immich-init-wait-server
24m                   Normal    Pulled                   Pod/immich-microservices-5f7598bf4d-tz68g      Container image "tccr.io/tccr/alpine:v3.21.2@sha256:b25393956a2e693711e9de978f5a884c88f51064d64bb48e7f699fbe75d430b6" already present on machine
24m                   Normal    Started                  Pod/immich-77f98ff5bd-4p5bv                    Started container immich-system-cnpg-wait
24m                   Normal    Started                  Pod/immich-77f98ff5bd-4p5bv                    Started container immich-system-redis-wait
24m                   Normal    Pulling                  Pod/immich-77f98ff5bd-4p5bv                    Pulling image "ghcr.io/immich-app/immich-server:v1.126.1@sha256:f8a3c78ec0a0ace20517acedaab9c2f3edcfc9b96e03080ba401acf55792470e"
23m                   Normal    Started                  Pod/immich-77f98ff5bd-4p5bv                    Started container immich
23m                   Normal    Created                  Pod/immich-77f98ff5bd-4p5bv                    Created container immich
23m                   Normal    Pulled                   Pod/immich-77f98ff5bd-4p5bv                    Successfully pulled image "ghcr.io/immich-app/immich-server:v1.126.1@sha256:f8a3c78ec0a0ace20517acedaab9c2f3edcfc9b96e03080ba401acf55792470e" in 42.495s (42.495s including waiting). Image size: 683340554 bytes.
23m                   Normal    Pulling                  Pod/immich-machinelearning-d45ff9cd7-72sgv     Pulling image "ghcr.io/immich-app/immich-machine-learning:v1.126.1-openvino@sha256:0eab7904f965ff76192c7679a00ad87527224d285b9ca4fec30cba0c401ec5e1"
23m                   Normal    Pulled                   Pod/immich-microservices-5f7598bf4d-tz68g      Container image "ghcr.io/immich-app/immich-server:v1.126.1@sha256:f8a3c78ec0a0ace20517acedaab9c2f3edcfc9b96e03080ba401acf55792470e" already present on machine
23m                   Normal    Started                  Pod/immich-microservices-5f7598bf4d-tz68g      Started container immich
23m                   Normal    Created                  Pod/immich-microservices-5f7598bf4d-tz68g      Created container immich
22m                   Normal    Pulled                   Pod/immich-machinelearning-d45ff9cd7-72sgv     Successfully pulled image "ghcr.io/immich-app/immich-machine-learning:v1.126.1-openvino@sha256:0eab7904f965ff76192c7679a00ad87527224d285b9ca4fec30cba0c401ec5e1" in 23.314s (23.314s including waiting). Image size: 392450786 bytes.
22m                   Normal    Created                  Pod/immich-machinelearning-d45ff9cd7-72sgv     Created container immich
22m                   Normal    Started                  Pod/immich-machinelearning-d45ff9cd7-72sgv     Started container immich
22m                   Normal    UpgradeSucceeded         HelmRelease/immich                             Helm upgrade succeeded for release immich/immich.v67 with chart [email protected]

Chart Logs

n/a?

Chart Configuration

values:
TZ: ${TZ}
ingress:
main:
enabled: true
hosts:
- host: immich.${DOMAIN_0}
paths:
- path: /
pathType: Prefix
integrations:
traefik:
enabled: true
certManager:
enabled: true
certificateIssuer: cert
homepage:
description: ""
enabled: true
group: Other
icon: ""
name: ""
widget:
custom:
key: ${IMMICH_KEY}
enabled: false
immich:
enable_ml: true
mlImageType: mlOpenvinoImage
log_level: log

workload:
  machinelearning:
    podSpec:
      containers:
        machinelearning:
          resources:
            limits:
              gpu.intel.com/i915: 1

Describe the bug

gpu resource limits defined in values don't show up in the cluster when deployed

To Reproduce

  1. request a gpu in values
  2. deploy
  3. check pod in dashboard; see no gpus

Expected Behavior

gpu is available to the pod

Screenshots

n/a?

Additional Context

I got some assistance in the discord from Alfi, and he said that it looks like tpl is overwriting the values, and that my magic phrase is "tpl merge". Also my talos version is 1.9, not 1.7.

I've read and agree with the following

  • I've checked all open and closed issues and my issue is not there.
  • I've understand TrueNAS SCALE Apps are deprecated and TrueCharts does not accept TrueNAS SCALE related issues
  • I've prefixed my issue title with [Chart-Name]
@WhiffyMuffinz WhiffyMuffinz added the bug Something isn't working label Feb 13, 2025
@alfi0812 alfi0812 changed the title fix[immich]: tpl merge not propagating resource limits fix[immich]: tpl not respecting container resources Feb 13, 2025
@alfi0812
Copy link
Collaborator

Immich uses a lot of tpl which is unnecessary and should be redone to values.
Tpl also causes the issues that they are done in a way that they overwrite and arent merged with values

@stavros-k stavros-k self-assigned this Feb 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stavros-k stavros-k

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@alfi0812 @stavros-k @WhiffyMuffinz