Connection Refused since latest update

[0xFEEDC0DE64]

The bug

Hi,
I get a connection refused from immich web since my latest docker pull
I have already tried running both immich containers with --net=host to avoid any problems with docker's tcp proxy but the issue seems to be that immich stopped listening on tcp ports.

immich web always was on port 3001, port 3003 seems to return a json:

$ curl http://localhost:3001
curl: (7) Failed to connect to localhost port 3001 after 0 ms: Couldn't connect to server
7 $ curl http://localhost:3002
curl: (7) Failed to connect to localhost port 3002 after 0 ms: Couldn't connect to server
7 $ curl http://localhost:3003
{"message":"Immich ML"}

I did do a lot of changes to the docker-compose since I dont want to run another database server or another redis server and I want to share my existing infrastructure (is also easier to backup for me).

If I did something wrong maybe you can pinpoint me in the right direction, but for now I dont know how to access immich web again.

The OS that Immich Server is running on

Archlinux

Version of Immich Server

release

Version of Immich Mobile App

no

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    network_mode: "host"  
#    ports:
#      - 2283:3001
#    depends_on:
#      - redis
#      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    network_mode: "host"  
    restart: always

#  redis:
#    container_name: immich_redis
#    image: docker.io/redis:6.2-alpine@sha256:328fe6a5822256d065debb36617a8169dbfbd77b797c525288e465f56c1d392b
#    healthcheck:
#      test: redis-cli ping || exit 1
#    restart: always

#  database:
#    container_name: immich_postgres
#    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
#    environment:
#      POSTGRES_PASSWORD: ${DB_PASSWORD}
#      POSTGRES_USER: ${DB_USERNAME}
#      POSTGRES_DB: ${DB_DATABASE_NAME}
#      POSTGRES_INITDB_ARGS: '--data-checksums'
#    volumes:
#      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
#    healthcheck:
#      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
#      interval: 5m
#      start_interval: 30s
#      start_period: 5m
#    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
#    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt2/Bilder
# The location where your database files are stored
#DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Europe/Vienna
IMMICH_HOST=127.0.0.1
#IMMICH_PORT=2283

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=<removed>
#DB_PASSWORD=<removed>

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=192.168.0.2
DB_USERNAME=immich
#DB_USERNAME=feedc0de
DB_DATABASE_NAME=immich
DB_VECTOR_EXTENSION=pgvector

REDIS_HOSTNAME=127.0.0.1
REDIS_PASSWORD=ieIF1Xyf7qVQ1E8Mqx55jOA4NCVNByWI

Reproduction steps

I guess will be hard since my postgres database cluster at home is not easy to reproduce :)

Relevant log output

$ docker-compose up
[+] Running 2/2
 ✔ Container immich_server            Recreated                                                                                                                                                                                                                                                                          0.3s 
 ✔ Container immich_machine_learning  Recreated                                                                                                                                                                                                                                                                          0.3s 
Attaching to immich_machine_learning, immich_server
immich_server            | Detected CPU Cores: 8
immich_machine_learning  | [07/04/24 19:44:19] INFO     Starting gunicorn 22.0.0                           
immich_machine_learning  | [07/04/24 19:44:19] INFO     Listening at: http://127.0.0.1:3003 (9)            
immich_machine_learning  | [07/04/24 19:44:19] INFO     Using worker: app.config.CustomUvicornWorker       
immich_machine_learning  | [07/04/24 19:44:19] INFO     Booting worker with pid: 10                        
immich_server            | Starting api worker
immich_server            | Starting microservices worker
immich_server            | [Nest] 7  - 07/04/2024, 7:44:26 PM     LOG [Microservices:EventRepository] Initialized websocket server
immich_machine_learning  | [07/04/24 19:44:26] INFO     Started server process [10]                        
immich_machine_learning  | [07/04/24 19:44:26] INFO     Waiting for application startup.                   
immich_machine_learning  | [07/04/24 19:44:26] INFO     Created in-memory cache with unloading after 300s  
immich_machine_learning  |                              of inactivity.                                     
immich_machine_learning  | [07/04/24 19:44:26] INFO     Initialized request thread pool with 8 threads.    
immich_machine_learning  | [07/04/24 19:44:26] INFO     Application startup complete.                      
immich_server            | [Nest] 17  - 07/04/2024, 7:44:26 PM     LOG [Api:EventRepository] Initialized websocket server

Additional information

No response

[mmomjian]

Remove IMMICH_HOST from your .env, that’s probably the issue. You’re telling it to only listen on the docker container internally.

[0xFEEDC0DE64]

Thanks for the tip, I have removed IMMICH_HOST (i have only added it for debugging to be honest), and it behaves now exactly the same

$ docker ps -a
CONTAINER ID   IMAGE                                                COMMAND                  CREATED              STATUS                          PORTS     NAMES
943ceec8d890   ghcr.io/immich-app/immich-server:release             "tini -- /bin/bash s…"   About a minute ago   Up About a minute (healthy)               immich_server
a81940f8310d   ghcr.io/immich-app/immich-machine-learning:release   "tini -- ./start.sh"     About a minute ago   Up About a minute (unhealthy)             immich_machine_learning
[feedc0de@flucky-server immich-app]$ docker logs -f immich_server
Detected CPU Cores: 8
Starting api worker
Starting microservices worker
[Nest] 7  - 07/05/2024, 2:58:44 PM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 17  - 07/05/2024, 2:58:44 PM     LOG [Api:EventRepository] Initialized websocket server

[0xFEEDC0DE64]

With log level set to verbose I still dont see which port immich even tries to listen on, the old port still is connection refused:

$ docker-compose up 
[+] Running 2/2
 ✔ Container immich_machine_learning  Recreated                                                                                                          1.1s 
 ✔ Container immich_server            Recreated                                                                                                          1.1s 
Attaching to immich_machine_learning, immich_server
immich_server            | DEBUG: cgroup v2 detected.
immich_server            | DEBUG: No CPU limits set.
immich_server            | Detected CPU Cores: 8
immich_machine_learning  | [07/05/24 15:02:57] INFO     Starting gunicorn 22.0.0                           
immich_machine_learning  | [07/05/24 15:02:57] INFO     Listening at: http://[::]:3003 (9)                 
immich_machine_learning  | [07/05/24 15:02:57] INFO     Using worker: app.config.CustomUvicornWorker       
immich_machine_learning  | [07/05/24 15:02:57] INFO     Booting worker with pid: 10                        
immich_machine_learning  | [07/05/24 15:02:58] DEBUG    Could not load ANN shared libraries, using ONNX:   
immich_machine_learning  |                              libmali.so: cannot open shared object file: No such
immich_machine_learning  |                              file or directory                                  
immich_server            | Starting api worker
immich_server            | Starting microservices worker
immich_machine_learning  | [07/05/24 15:03:04] INFO     Started server process [10]                        
immich_machine_learning  | [07/05/24 15:03:04] INFO     Waiting for application startup.                   
immich_machine_learning  | [07/05/24 15:03:04] INFO     Created in-memory cache with unloading after 300s  
immich_machine_learning  |                              of inactivity.                                     
immich_machine_learning  | [07/05/24 15:03:04] INFO     Initialized request thread pool with 8 threads.    
immich_machine_learning  | [07/05/24 15:03:04] DEBUG    Checking for inactivity...                         
immich_machine_learning  | [07/05/24 15:03:04] INFO     Application startup complete.                      
immich_server            | [Nest] 7  - 07/05/2024, 3:03:04 PM     LOG [Microservices:EventRepository] Initialized websocket server
immich_server            | [Nest] 17  - 07/05/2024, 3:03:05 PM     LOG [Api:EventRepository] Initialized websocket server
immich_machine_learning  | [07/05/24 15:03:14] DEBUG    Checking for inactivity...                         
immich_machine_learning  | [07/05/24 15:03:24] DEBUG    Checking for inactivity...                         
immich_machine_learning  | [07/05/24 15:03:34] DEBUG    Checking for inactivity...                         
immich_machine_learning  | [07/05/24 15:03:44] DEBUG    Checking for inactivity...     

If immich doesnt provide any webinterface anymore, is there an easy way to migrate to ente or nextcloud?

[mmomjian]

The Immich files are all stored in UPLOAD_LOCATION.

This is definitely something to do with your very weird setup. You are also missing object files: libmali.so, which indicates some kind of underlying problem with your docker image / merger layers. I don't know if Immich is tested / supported to work with network_mode: host. I would recommend removing all of those.

Try curling localhost:2283 to see if that does anything

[0xFEEDC0DE64]

I only added all the weird things to be able to debug the network issues, before the failed update it was all connected using the docker networks.
But the docker tcp proxy started to close down connections and i started investigating, and found out that immich stopped listening on the port
Then I switched to network=host to debug this and in fact it does not listen anymore

[0xFEEDC0DE64]

I could solve the issue, I was debugging with strace and ptrace and found out it was stalled waiting on SQL queries being done, CREATE TABLE, CREATE INDEX and so on!
So I let it run over night and today it semms like it finished and is reachable again on :3001
The log output is a bit useless during this phase as it just looks like it doesnt do anything.

Thanks for trying to help fix this

[shei-pi]

I'm having the same issue but letting it run overnight didn't help. It may be that my server is struggling to get it done, but I left it for a week and still nothing. Can you share some details on how did you debug this?

[xicraa]

You need to change the port. This is, change the text "2283:3001" to this text: "2283:2283". Then wait about 30 seconds.

For me, it worked.

Best regards.

[xicraa]

You need to change the port. This is, change the text "2283:3001" to this text: "2283:2283". Then wait about 30 seconds.

For me, it worked.

Best regards.

[jev-on]

You need to change the port. This is, change the text "2283:3001" to this text: "2283:2283". Then wait about 30 seconds.

For me, it worked.

Best regards.

Thanks, for posting this. I was using a docker-compose.yml I created some time back and at that point the port mapping was different. I was able to get Immich up and running again.