Error 500 - server crashes after 8-10 hours

[tk421storm]

The bug

server becomes unresponsive after 8-10 hours with this bug in the UI:

Error: 500 (undefined)

Stacktrace
Error: Error: 500
at Object.he [as ok] (http://192.168.1.5:2283/_app/immutable/chunks/fetch-client.Dt8dFg_1.js:1:2872)
at async Promise.all (index 0)
at async n (http://192.168.1.5:2283/_app/immutable/chunks/server-config.store.DegwLTU0.js:1:515)
at async o (http://192.168.1.5:2283/_app/immutable/chunks/server.DfmgLEeR.js:1:189)
at async Wi (http://192.168.1.5:2283/_app/immutable/nodes/0.DUxp1LQK.js:1:2069)
at async Ue (http://192.168.1.5:2283/_app/immutable/chunks/entry.Cv4Sd8Jo.js:1:15711)

and this bug repeated many times in the logs:

immich_server | [Nest] 18 - 01/06/2025, 11:51:12 AM ERROR [Api:ErrorInterceptor~pczw6klq] Unknown error: error: could not open file "global/pg_filenode.map": Permission denied

The OS that Immich Server is running on

Ubuntu 20.04 LTS

Version of Immich Server

v1.119.1

Version of Immich Mobile App

v1.119.1

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2ba50e1ac3a0ea17b736ce9db2b0a9f6f8b85d4c27d5f5accc6a416d8f42c6d5
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command:
      [
        'postgres',
        '-c',
        'shared_preload_libraries=vectors.so',
        '-c',
        'search_path="$$user", public, vectors',
        '-c',
        'logging_collector=on',
        '-c',
        'max_wal_size=2GB',
        '-c',
        'shared_buffers=512MB',
        '-c',
        'wal_compression=on',
      ]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/torgo/Photos/immich/photos
# The location where your database files are stored
DB_DATA_LOCATION=/torgo/Photos/immich/data

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=v1.119.1

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=XXXXXXXXXX

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. start servers with sudo docker compose up or sudo docker compose start
2. wait 8-10 hours

Relevant log output

No response

Additional information

No response

[alextran1502]

DB_DATA_LOCATION=/torgo/Photos/immich/data

is that a network mount? Looks like your DB is corrupted

[tk421storm]

it's a local mount, made new for this instance. been attempting to upgrade to 1.119 for a few days - I want the database stored at a local path rather than in the docker volume. I delete everything under "data" in that directory first then run docker compose --force-rebuild. Should I allow immich to make that containing folder "data" itself?

[bo0tzz]

The timing suggests to me that something else might be messing with the permissions on the folder (otherwise I'd expect it to break immediately). Letting the data folder be created automatically seems worth a shot though, that should ensure that at least initially the permissions are correct.

[tk421storm]

confirmed that oddly the owner for that folder (and that folder only) has changed (from 1000:2000 to 999:999). will delete the "data" folder and recreate server to see if it occurs again

[tk421storm]

yup, there was a cronjob changing permissions every night. fixed, thank you!