Clean install 1.130.2 causes microservices worker error: TypeError: Cannot read properties of undefined (reading 'replace')

[o10s]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

Created a new issue due to a comment from @bo0tzz on discussion topic #17152

The OS that Immich Server is running on

Kubernetes RKE2 - On top of ubuntu server 24.10

Version of Immich Server

v1.130.2

Version of Immich Mobile App

v1.130.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

I use a custom helm, the following is my values file should provide all information required for debugging I suppose

immichPostgres:
  applicationName: immich-psql
  deployment: 
    enabled: false
  service:
    enabled: false
  externalSecret:
    enabled: true
    secretStore:
      name: bitwarden-secretsmanager
      kind: ClusterSecretStore
    files: 
      credentials:
        data:
          postgres-password:
            remoteRef:
              key: <REDACTED>
          password:
            remoteRef:
              key: <REDACTED>
          replication-password:
            remoteRef:
              key: <REDACTED>

immichServer:
  applicationName: immich-server
  deployment: 
    enabled: true
    image:
      repository: ghcr.io/immich-app/immich-server
      tag: v1.130.2
    env:
      IMMICH_CONFIG_FILE:
        value: /config/immich-config.yaml
      DB_HOSTNAME:
        value: immich-postgresql
      DB_USERNAME:
        value: immich
      DB_DATABASE_NAME:
        value: immich
      DB_PASSWORD:
        valueFrom:
          secretKeyRef:
            name: immich-psql-credentials
            key: postgres-password
      IMMICH_MACHINE_LEARNING_URL:
        value: http://immich-machine-learning:3003
      REDIS_HOSTNAME:
        value: immich-redis-headless

    resources:
      limits:
        cpu: 4000m
        memory: 10Gi
      requests:
        cpu: 1000m
        memory: 512Mi
    containerSecurityContext:
      readOnlyRootFilesystem: false
      runAsNonRoot: false
    volumes:
      nfs-photos:
        nfs:
          server: <REDACTED>
          path: <REDACTED>
      dev-dri:
        hostPath:
          path: /dev/dri
          type: Directory
    volumeMounts:
      nfs-photos:
        mountPath: /usr/src/app/upload
      dev-dri:
        mountPath: /dev/dri
  persistence:
    enabled: true
    mountPVC: true
    mountPath: "/config"
    storageSize: 10Gi
  service:
    type: ClusterIP
    ports:
      - name: http
        port: 2283
        targetPort: 2283
  ingress:
    enabled: true
    hosts:
      - host: immich.<REDACTED>
        paths:
          - path: /
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-staging
    tls:
      - secretName: immich-tls
        hosts:
          - immich.<REDACTED>

immichMachineLearning:
  applicationName: immich-machine-learning
  deployment: 
    enabled: true
    image:
      repository: ghcr.io/immich-app/immich-machine-learning
      tag: v1.130.2-openvino
    env:
      TRANSFORMERS_CACHE:
        value: /cache
      DB_HOSTNAME:
        value: immich-postgresql.immich.svc.cluster.local
      DB_USERNAME:
        value: immich
      DB_DATABASE_NAME:
        value: immich
      DB_PASSWORD:
        valueFrom:
          secretKeyRef:
            name: immich-psql-credentials
            key: postgres-password
      IMMICH_MACHINE_LEARNING_URL:
        value: http://immich-machine-learning:3003
      REDIS_HOSTNAME:
        value: immich-redis-headless
    containerSecurityContext:
      readOnlyRootFilesystem: false
      runAsNonRoot: false
    volumes:
      dev-dri:
        hostPath:
          path: /dev/dri
          type: Directory
      dev-bus-usb:
        hostPath:
          path: /dev/bus/usb
          type: Directory
    volumeMounts:
      dev-dri:
        mountPath: /dev/dri
      dev-bus-usb:
        mountPath: /dev/bus/usb
  service:
    enabled: true
    type: ClusterIP
    ports:
      - name: http
        port: 3003
        targetPort: 3003

redis:
  architecture: standalone
  auth:
    enabled: false

postgresql:
  enabled: true
  image:
    repository: tensorchord/pgvecto-rs
    tag: pg14-v0.2.0@sha256:739cdd626151ff1f796dc95a6591b55a714f341c737e27f045019ceabf8e8c52
  global:
    postgresql:
      auth:
        username: immich
        database: immich
        existingSecret: "immich-psql-credentials"
        secretKeys:
          adminPasswordKey: postgres-password
          userPasswordKey: password
          replicationPasswordKey: replication-password
  primary:
    containerSecurityContext:
      readOnlyRootFilesystem: false
    initdb:
      scripts:
        create-extensions.sql: |
          CREATE EXTENSION cube;
          CREATE EXTENSION earthdistance;
          CREATE EXTENSION vectors;

Your .env content

All envs are defined above in the values file

Reproduction steps

1. Launch with the given values
2. Wait
3. Crash of the server
   ...

Relevant log output

Initializing Immich v1.130.2
Detected CPU Cores: 4
Starting api worker
Starting microservices worker
[Nest] 7  - 03/27/2025, 3:35:16 PM     LOG [Microservices:EventRepository] Initialized websocket server
microservices worker error: TypeError: Cannot read properties of undefined (reading 'replace'), stack: TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (/usr/src/app/node_modules/postgres/cjs/src/connection.js:389:48)
    at errored (/usr/src/app/node_modules/postgres/cjs/src/connection.js:384:17)
    at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:318:9)
    at Socket.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:561:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
    at Readable.push (node:internal/streams/readable:392:5)
    at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
microservices worker exited with code 1
Killing api process

Additional information

It could be due to a bug in alpine as described here: #17152
Or it could be due to a communication issue but I don't see how...

[alextran1502]

Can you please check the database log?

[o10s]

@alextran1502 The logs are as follows (I do see the chmod issue which can be a problem):

chmod: changing permissions of '/var/run/postgresql': Operation not permitted
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /bitnami/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok
Success. You can now start the database server using:
    pg_ctl -D /bitnami/postgresql/data -l logfile start
waiting for server to start....2025-03-27 15:49:11.147 UTC [39] LOG:  redirecting log output to logging collector process
2025-03-27 15:49:11.147 UTC [39] HINT:  Future log output will appear in directory "log".
 done
server started
CREATE DATABASE
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/create-extensions.sql
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
waiting for server to shut down.... done
server stopped
PostgreSQL init process complete; ready for start up.
2025-03-27 15:49:12.361 UTC [1] LOG:  redirecting log output to logging collector process
2025-03-27 15:49:12.361 UTC [1] HINT:  Future log output will appear in directory "log".

And an image of the deployments in ArgoCD

They are all in the same namespace, this postgres instance is only used for immich.

[o10s]

@alextran1502 In the .log files (wasn't shown in logs output as per stated in the output above) I saw that it was a wrong password issue!

(thanks for pointing me in the right direction!)

[o10s]

After solving the DB password issue I have another error @alextran1502
I receive in my DB the kill signal (for unknown reason)

2025-03-27 16:06:14.521 UTC [39] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-03-27 16:06:14.526 UTC [41] LOG:  database system was shut down at 2025-03-27 16:06:14 UTC
2025-03-27 16:06:14.529 UTC [39] LOG:  database system is ready to accept connections
2025-03-27 16:06:15.525 UTC [39] LOG:  received fast shutdown request
2025-03-27 16:06:15.528 UTC [39] LOG:  aborting any active transactions
2025-03-27 16:06:15.619 UTC [39] LOG:  background worker "logical replication launcher" (PID 50) exited with exit code 1
2025-03-27 16:06:15.619 UTC [45] LOG:  shutting down
2025-03-27 16:06:15.693 UTC [39] LOG:  database system is shut down

This happens when Immich initializes and tries to insert some stuff into the DB (such as countries etc)

[o10s]

For future users it was due to default too small resources on postgresql all good now !

[alextran1502]

😆

[kamasylvia]

For future users it was due to default too small resources on postgresql all good now !

I have the same problem. How do u solve it?

[o10s]

For future users it was due to default too small resources on postgresql all good now !

I have the same problem. How do u solve it?

in postgresql from bitnani you can give it more resources by default it starts with the nano profile I went up to large (it's basically a resources default profile)