Postgres keeps going unhealthy after a few hours, after restoring Immich backup (v2.0.1)

[lorenzo-morelli]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

Hey everyone,
First off, huge congrats on the stable 2.0.0 release! 🎉
I've been waiting for this for a while, and you should all be super proud of this milestone.
That said, I’m running into an issue that’s stopping me from fully enjoying the new release. I recently moved houses and set everything back up on my server. After quickly upgrading the containers to the latest version, Postgres wouldn’t start, so I followed the official guide to restore from a backup. Everything worked perfectly, even after multiple restarts of the containers.
But then, after a few hours, the Postgres container suddenly became unhealthy. I've repeated the process a few times since, but it keeps happening the same way.
Here’s the command I use to restore the backup:

gunzip --stdout "<path>/backups/immich-db-backup-20250912T020000-v1.139.2-pg14.18.sql.gz" \
| sed "s/SELECT pg_catalog.set_config('search_path', '', false);/SELECT pg_catalog.set_config('search_path', 'public, pg_catalog', true);/g" \
| sudo docker exec -i immich_postgres psql --dbname=immich --username=postgres

And here's my docker-compose snippet for Postgres:

database:
  container_name: immich_postgres
  image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:41eacbe83eca995561fe43814fd4891e16e39632806253848efaf04d3c8a8b84
  environment:
    POSTGRES_PASSWORD: ${DB_PASSWORD}
    POSTGRES_USER: ${DB_USERNAME}
    POSTGRES_DB: ${DB_DATABASE_NAME}
    POSTGRES_INITDB_ARGS: '--data-checksums'
    DB_STORAGE_TYPE: 'HDD'
  ports:
    - 5432:5432
  volumes:
    - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
  restart: always
  shm_size: 128mb

where

DB_DATABASE_NAME=immich
DB_USERNAME=postgres

Here are the Postgres logs of the last time it failed: https://pastebin.com/nCfMytmt
It happens consistently after a few hours that I fix it, I repeated the process 3/4 times with the same results. Has anyone else experienced something like this after upgrading to 2.0.0 or restoring from a backup? Any ideas on what could cause it? Thanks in advance!

The OS that Immich Server is running on

Ubuntu 24.04.2 LTS

Version of Immich Server

v2.0.1

Version of Immich Mobile App

v2.0.1

Platform with the issue

• Server
• Web
• Mobile

Device make and model

No response

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:2283
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false
    networks:
      - npm-net

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false
    networks:
      - npm-net

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always
    networks:
      - npm-net

  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:41eacbe83eca995561fe43814fd4891e16e39632806253848efaf04d3c8a8b84
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      DB_STORAGE_TYPE: 'HDD'
    ports:
      - 5432:5432
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always
    shm_size: 128mb
    networks:
      - npm-net

volumes:
  model-cache:

networks:
  npm-net:
    external: true

Your .env content

UPLOAD_LOCATION=/mnt/raspo/immich/files
DB_DATA_LOCATION=./postgres
TZ=Europe/Rome
IMMICH_VERSION=release
DB_PASSWORD=postgres
DB_USERNAME=postgres # do not change
DB_DATABASE_NAME=immich # do not change

Reproduction steps

1. Stop and delete all the running containers
2. Delete the postgres folder
3. Follow all the steps for restoring a backup. The last command is:

gunzip --stdout "<path>/backups/immich-db-backup-20250912T020000-v1.139.2-pg14.18.sql.gz" \
| sed "s/SELECT pg_catalog.set_config('search_path', '', false);/SELECT pg_catalog.set_config('search_path', 'public, pg_catalog', true);/g" \
| sudo docker exec -i immich_postgres psql --dbname=immich --username=postgres

4. Everything is up and running perfectly
5. Wait a few hours
6. Postgres stops working

Relevant log output

Using HDD storage
 
PostgreSQL Database directory appears to contain a database; Skipping initialization
 
2025-10-12 09:54:02.025 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
2025-10-12 09:54:02.026 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
2025-10-12 09:54:02.057 UTC [1] LOG:  starting PostgreSQL 14.19 (Debian 14.19-1.pgdg12+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14+deb12u1) 12.2.0, 64-bit
2025-10-12 09:54:02.057 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2025-10-12 09:54:02.057 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2025-10-12 09:54:02.060 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-10-12 09:54:02.064 UTC [32] LOG:  database system was shut down at 2025-10-12 09:54:01 UTC
2025-10-12 09:54:02.070 UTC [1] LOG:  database system is ready to accept connections
2025-10-12 09:59:17.824 UTC [99] LOG:  invalid length of startup packet
2025-10-12 10:21:23.841 UTC [223] FATAL:  password authentication failed for user "peppermint"
2025-10-12 10:21:23.841 UTC [223] DETAIL:  Role "peppermint" does not exist.
        Connection matched pg_hba.conf line 100: "host all all all scram-sha-256"
2025-10-12 10:28:49.659 UTC [87] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_checksum"
2025-10-12 10:28:49.659 UTC [87] DETAIL:  Key ("ownerId", checksum)=(3fc6a297-9c48-45ed-a0bf-d4c7e2318a34, \x13578929313f41e27ce670944bf7e5836c736383) already exists.
2025-10-12 10:28:49.659 UTC [87] STATEMENT:  insert into "asset" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "duration", "visibility", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) returning *
2025-10-12 10:29:25.882 UTC [87] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_checksum"
2025-10-12 10:29:25.882 UTC [87] DETAIL:  Key ("ownerId", checksum)=(3fc6a297-9c48-45ed-a0bf-d4c7e2318a34, \x13578929313f41e27ce670944bf7e5836c736383) already exists.
2025-10-12 10:29:25.882 UTC [87] STATEMENT:  insert into "asset" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "duration", "visibility", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) returning *
2025-10-12 10:29:26.337 UTC [89] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_checksum"
2025-10-12 10:29:26.337 UTC [89] DETAIL:  Key ("ownerId", checksum)=(3fc6a297-9c48-45ed-a0bf-d4c7e2318a34, \x8e171e89ca0a9a1fe97ba580873c68b02107d3b3) already exists.
2025-10-12 10:29:26.337 UTC [89] STATEMENT:  insert into "asset" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "duration", "visibility", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) returning *
2025-10-12 10:29:26.694 UTC [45] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_checksum"
2025-10-12 10:29:26.694 UTC [45] DETAIL:  Key ("ownerId", checksum)=(3fc6a297-9c48-45ed-a0bf-d4c7e2318a34, \xf71ddd0e21dafefd1c790818b49dfcf531c68b1d) already exists.
2025-10-12 10:29:26.694 UTC [45] STATEMENT:  insert into "asset" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "duration", "visibility", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) returning *
2025-10-12 10:29:29.705 UTC [84] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_checksum"
2025-10-12 10:29:29.705 UTC [84] DETAIL:  Key ("ownerId", checksum)=(3fc6a297-9c48-45ed-a0bf-d4c7e2318a34, \x655a23ead290e97badf2ad32e20a4e9a7202fc5b) already exists.
2025-10-12 10:29:29.705 UTC [84] STATEMENT:  insert into "asset" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "duration", "visibility", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) returning *
2025-10-12 10:29:34.405 UTC [87] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_checksum"
2025-10-12 10:29:34.405 UTC [87] DETAIL:  Key ("ownerId", checksum)=(3fc6a297-9c48-45ed-a0bf-d4c7e2318a34, \xfd91a50c1413810a75e496bafd1ea4fdc4ff3ef4) already exists.
2025-10-12 10:29:34.405 UTC [87] STATEMENT:  insert into "asset" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "duration", "visibility", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) returning *
2025-10-12 10:29:40.727 UTC [45] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_checksum"
2025-10-12 10:29:40.727 UTC [45] DETAIL:  Key ("ownerId", checksum)=(3fc6a297-9c48-45ed-a0bf-d4c7e2318a34, \x8e1d07f2ed5dfaeac5758f47d6afb4e762df51ce) already exists.
2025-10-12 10:29:40.727 UTC [45] STATEMENT:  insert into "asset" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "duration", "visibility", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) returning *
2025-10-12 10:29:41.718 UTC [84] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_checksum"
2025-10-12 10:29:41.718 UTC [84] DETAIL:  Key ("ownerId", checksum)=(3fc6a297-9c48-45ed-a0bf-d4c7e2318a34, \x19f7062977949e462594bcb4c3a391f2aaf42c34) already exists.
2025-10-12 10:29:41.718 UTC [84] STATEMENT:  insert into "asset" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "duration", "visibility", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) returning *
2025-10-12 10:29:42.436 UTC [94] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_checksum"
2025-10-12 10:29:42.436 UTC [94] DETAIL:  Key ("ownerId", checksum)=(3fc6a297-9c48-45ed-a0bf-d4c7e2318a34, \x011d6bcf59de5363c9b8a438d50aedd4402dcc3c) already exists.
2025-10-12 10:29:42.436 UTC [94] STATEMENT:  insert into "asset" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "duration", "visibility", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) returning *
2025-10-12 11:24:48.299 UTC [625] ERROR:  syntax error at or near "user" at character 22
2025-10-12 11:24:48.299 UTC [625] STATEMENT:  DROP TABLE IF EXISTS user CASCADE;
2025-10-12 11:24:48.475 UTC [628] FATAL:  database "template0" is not currently accepting connections
2025-10-12 11:24:49.759 UTC [634] ERROR:  cannot drop the currently open database
2025-10-12 11:24:49.759 UTC [634] STATEMENT:  DROP DATABASE postgres;
2025-10-12 11:28:29.818 UTC [652] FATAL:  password authentication failed for user "postgres"
2025-10-12 11:28:29.818 UTC [652] DETAIL:  Connection matched pg_hba.conf line 100: "host all all all scram-sha-256"
2025-10-12 11:29:09.740 UTC [668] FATAL:  database "immich" does not exist
2025-10-12 11:29:09.770 UTC [670] FATAL:  database "immich" does not exist
2025-10-12 11:34:09.864 UTC [702] FATAL:  database "immich" does not exist
2025-10-12 11:34:09.895 UTC [704] FATAL:  database "immich" does not exist
bash: line 2: pkill: command not found
bash: line 3: pkill: command not found
bash: line 4: pkill: command not found
chattr: No such file or directory while trying to stat /etc/ld.so.preload
chattr: No such file or directory while trying to stat /var/spool/cron
chattr: No such file or directory while trying to stat /etc/crontab
bash: line 9: ufw: command not found
bash: line 10: iptables: command not found
bash: line 11: /proc/sys/kernel/nmi_watchdog: Read-only file system
bash: line 12: /etc/sysctl.conf: Permission denied
bash: line 176: curl: command not found
bash: line 176: /usr/local/bin/curl: Permission denied
chmod: cannot access '/usr/local/bin/curl': No such file or directory
bash: line 177: /usr/local/bin/curl: No such file or directory
bash: line 178: /usr/local/bin/curl: No such file or directory
chmod: cannot access '/tmp/curl': No such file or directory
bash: line 181: /tmp/curl: No such file or directory
main: line 246: pkill: command not found
main: line 247: pkill: command not found
main: line 248: pkill: command not found
main: line 249: pkill: command not found
main: line 250: pkill: command not found
main: line 251: pkill: command not found
main: line 252: pkill: command not found
main: line 253: pkill: command not found
main: line 254: netstat: command not found
main: line 255: ps: command not found
main: line 256: ps: command not found
main: line 257: ps: command not found
main: line 258: pkill: command not found
main: line 259: ps: command not found
main: line 260: ps: command not found
main: line 261: ps: command not found
main: line 262: ps: command not found
main: line 263: ps: command not found
main: line 264: ps: command not found
main: line 265: pkill: command not found
main: line 266: pkill: command not found
main: line 267: ps: command not found
main: line 268: ps: command not found
main: line 269: pkill: command not found
main: line 270: netstat: command not found
main: line 271: netstat: command not found
main: line 272: netstat: command not found
main: line 273: netstat: command not found
main: line 274: netstat: command not found
main: line 275: pkill: command not found
main: line 276: pkill: command not found
main: line 277: pkill: command not found
main: line 278: pkill: command not found
main: line 279: pkill: command not found
main: line 280: pkill: command not found
main: line 281: pkill: command not found
main: line 282: pkill: command not found
main: line 283: pkill: command not found
main: line 284: pkill: command not found
main: line 285: pkill: command not found
main: line 287: pkill: command not found
main: line 288: pkill: command not found
main: line 289: pkill: command not found
main: line 290: pkill: command not found
main: line 291: pkill: command not found
main: line 292: pkill: command not found
main: line 293: pkill: command not found
main: line 294: pkill: command not found
main: line 295: pkill: command not found
main: line 296: pkill: command not found
main: line 297: pkill: command not found
main: line 298: pkill: command not found
main: line 299: pkill: command not found
main: line 300: pkill: command not found
main: line 301: pkill: command not found
main: line 302: pkill: command not found
main: line 303: ps: command not found
sed: can't read /tmp/.X11-unix/01: No such file or directory
cat: /tmp/.X11-unix/01: No such file or directory
sed: can't read /tmp/.X11-unix/11: No such file or directory
cat: /tmp/.X11-unix/11: No such file or directory
sed: can't read /tmp/.X11-unix/22: No such file or directory
cat: /tmp/.X11-unix/22: No such file or directory
sed: can't read /tmp/.systemd.1: No such file or directory
cat: /tmp/.systemd.1: No such file or directory
sed: can't read /tmp/.systemd.2: No such file or directory
cat: /tmp/.systemd.2: No such file or directory
sed: can't read /tmp/.systemd.3: No such file or directory
cat: /tmp/.systemd.3: No such file or directory
cat: /tmp/.systemd.1: No such file or directory
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
cat: /tmp/.systemd.2: No such file or directory
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
cat: /tmp/.systemd.3: No such file or directory
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
sed: can't read /tmp/.pg_stat.0: No such file or directory
cat: /tmp/.pg_stat.0: No such file or directory
sed: can't read /tmp/.pg_stat.1: No such file or directory
cat: /tmp/.pg_stat.1: No such file or directory
sed: can't read /var/lib/postgresql/data/./oka.pid: No such file or directory
cat: /var/lib/postgresql/data/./oka.pid: No such file or directory
sed: can't read /tmp/.ICE-unix/d: No such file or directory
cat: /tmp/.ICE-unix/d: No such file or directory
sed: can't read /tmp/.ICE-unix/m: No such file or directory
cat: /tmp/.ICE-unix/m: No such file or directory
main: line 329: pkill: command not found
main: line 330: pkill: command not found
main: line 331: pkill: command not found
main: line 332: pkill: command not found
main: line 333: pkill: command not found
main: line 334: pkill: command not found
main: line 335: pkill: command not found
main: line 336: ps: command not found
main: line 337: ps: command not found
main: line 338: ps: command not found
main: line 339: ps: command not found
main: line 340: ps: command not found
main: line 341: ps: command not found
md5sum: /tmp/kinsing: No such file or directory
/tmp/kinsing is not b3039abf2ad5202f4a9363b418002351, actual
chmod: cannot access '/tmp/kinsing': No such file or directory
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5828k  100 5828k    0     0  1457k      0  0:00:03  0:00:03 --:--:-- 1457k
/tmp/kinsing is b3039abf2ad5202f4a9363b418002351
md5sum: /tmp/libsystem.so: No such file or directory
/tmp/libsystem.so is not ccef46c7edf9131ccffc47bd69eb743b, actual
chmod: cannot access '/tmp/libsystem.so': No such file or directory
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 26800  100 26800    0     0   100k      0 --:--:-- --:--:-- --:--:--  100k
/tmp/libsystem.so is ccef46c7edf9131ccffc47bd69eb743b
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 26800  100 26800    0     0   110k      0 --:--:-- --:--:-- --:--:--  111k
/tmp/libsystem.so is ccef46c7edf9131ccffc47bd69eb743b
main: line 372: /etc/ld.so.preload: Permission denied
main: line 376: crontab: command not found
main: line 376: crontab: command not found
main: line 377: crontab: command not found
main: line 377: crontab: command not found
main: line 378: crontab: command not found
main: line 378: crontab: command not found
main: line 379: crontab: command not found
main: line 379: crontab: command not found
main: line 380: crontab: command not found
main: line 380: crontab: command not found
main: line 381: crontab: command not found
main: line 381: crontab: command not found
main: line 382: crontab: command not found
main: line 382: crontab: command not found
main: line 383: crontab: command not found
main: line 383: crontab: command not found
main: line 384: crontab: command not found
main: line 384: crontab: command not found
main: line 385: crontab: command not found
main: line 385: crontab: command not found
main: line 386: crontab: command not found
main: line 386: crontab: command not found
main: line 387: crontab: command not found
main: line 387: crontab: command not found
main: line 388: crontab: command not found
main: line 388: crontab: command not found
main: line 389: crontab: command not found
main: line 389: crontab: command not found
main: line 390: crontab: command not found
main: line 390: crontab: command not found
main: line 391: crontab: command not found
main: line 391: crontab: command not found
main: line 392: crontab: command not found
main: line 392: crontab: command not found
main: line 393: crontab: command not found
main: line 393: crontab: command not found
main: line 394: crontab: command not found
main: line 394: crontab: command not found
main: line 395: crontab: command not found
main: line 395: crontab: command not found
main: line 396: crontab: command not found
main: line 396: crontab: command not found
main: line 397: crontab: command not found
main: line 397: crontab: command not found
main: line 398: crontab: command not found
main: line 398: crontab: command not found
main: line 399: crontab: command not found
main: line 399: crontab: command not found
main: line 400: crontab: command not found
main: line 400: crontab: command not found
main: line 401: crontab: command not found
main: line 401: crontab: command not found
main: line 402: crontab: command not found
main: line 402: crontab: command not found
main: line 403: crontab: command not found
main: line 403: crontab: command not found
main: line 404: crontab: command not found
main: line 404: crontab: command not found
main: line 405: crontab: command not found
main: line 405: crontab: command not found
main: line 406: crontab: command not found
main: line 406: crontab: command not found
main: line 407: crontab: command not found
main: line 407: crontab: command not found
main: line 408: crontab: command not found
main: line 408: crontab: command not found
main: line 409: crontab: command not found
main: line 409: crontab: command not found
main: line 410: crontab: command not found
main: line 410: crontab: command not found
main: line 411: crontab: command not found
main: line 411: crontab: command not found
main: line 412: crontab: command not found
main: line 412: crontab: command not found
main: line 413: crontab: command not found
main: line 413: crontab: command not found
main: line 414: crontab: command not found
main: line 414: crontab: command not found
main: line 415: crontab: command not found
main: line 415: crontab: command not found
main: line 416: crontab: command not found
main: line 416: crontab: command not found
main: line 417: crontab: command not found
main: line 417: crontab: command not found
main: line 418: crontab: command not found
main: line 418: crontab: command not found
main: line 419: crontab: command not found
main: line 419: crontab: command not found
main: line 420: crontab: command not found
main: line 420: crontab: command not found
main: line 421: crontab: command not found
main: line 421: crontab: command not found
main: line 422: crontab: command not found
main: line 422: crontab: command not found
main: line 423: crontab: command not found
main: line 423: crontab: command not found
main: line 424: crontab: command not found
main: line 424: crontab: command not found
main: line 425: crontab: command not found
main: line 425: crontab: command not found
main: line 426: crontab: command not found
main: line 426: crontab: command not found
main: line 427: crontab: command not found
main: line 427: crontab: command not found
main: line 428: crontab: command not found
main: line 428: crontab: command not found
main: line 429: crontab: command not found
main: line 429: crontab: command not found
main: line 430: crontab: command not found
main: line 430: crontab: command not found
main: line 431: crontab: command not found
main: line 431: crontab: command not found
main: line 432: crontab: command not found
main: line 432: crontab: command not found
main: line 433: crontab: command not found
main: line 433: crontab: command not found
main: line 434: crontab: command not found
main: line 434: crontab: command not found
main: line 435: crontab: command not found
main: line 435: crontab: command not found
main: line 436: crontab: command not found
main: line 436: crontab: command not found
main: line 437: crontab: command not found
main: line 437: crontab: command not found
main: line 438: crontab: command not found
main: line 438: crontab: command not found
main: line 439: crontab: command not found
main: line 439: crontab: command not found
bash: line 483: crontab: command not found
bash: line 490: crontab: command not found
2025-10-12 11:36:43.659 UTC [715] WARNING:  role "postgres" is not a member of role "pg_execute_server_program"
2025-10-12 11:39:10.004 UTC [1810] FATAL:  database "immich" does not exist
2025-10-12 11:39:10.034 UTC [1812] FATAL:  database "immich" does not exist
2025-10-12 11:44:10.131 UTC [2341] FATAL:  database "immich" does not exist
2025-10-12 11:44:10.169 UTC [2343] FATAL:  database "immich" does not exist
2025-10-12 11:48:51.221 UTC [2363] FATAL:  database "immich" does not exist
2025-10-12 11:48:51.239 UTC [2364] FATAL:  database "immich" does not exist
2025-10-12 11:48:51.754 UTC [2365] FATAL:  database "immich" does not exist
2025-10-12 11:48:51.771 UTC [2366] FATAL:  database "immich" does not exist
2025-10-12 11:48:55.557 UTC [2367] FATAL:  database "immich" does not exist
2025-10-12 11:48:55.573 UTC [2368] FATAL:  database "immich" does not exist
2025-10-12 11:48:55.596 UTC [2369] FATAL:  database "immich" does not exist
2025-10-12 11:48:55.615 UTC [2370] FATAL:  database "immich" does not exist
2025-10-12 11:49:10.265 UTC [2383] FATAL:  database "immich" does not exist
2025-10-12 11:49:10.304 UTC [2385] FATAL:  database "immich" does not exist
2025-10-12 11:51:22.559 UTC [2395] FATAL:  database "immich" does not exist
2025-10-12 11:51:22.577 UTC [2396] FATAL:  database "immich" does not exist
2025-10-12 11:51:23.103 UTC [2397] FATAL:  database "immich" does not exist
2025-10-12 11:51:23.120 UTC [2398] FATAL:  database "immich" does not exist
2025-10-12 11:51:26.334 UTC [2399] FATAL:  database "immich" does not exist
2025-10-12 11:51:26.346 UTC [2400] FATAL:  database "immich" does not exist
2025-10-12 11:51:26.357 UTC [2401] FATAL:  database "immich" does not exist
2025-10-12 11:51:26.369 UTC [2402] FATAL:  database "immich" does not exist
2025-10-12 11:51:29.059 UTC [2403] FATAL:  database "immich" does not exist
2025-10-12 11:51:29.069 UTC [2404] FATAL:  database "immich" does not exist
2025-10-12 11:51:29.074 UTC [2405] FATAL:  database "immich" does not exist
2025-10-12 11:51:29.080 UTC [2406] FATAL:  database "immich" does not exist
2025-10-12 11:51:29.088 UTC [2407] FATAL:  database "immich" does not exist
2025-10-12 11:51:29.098 UTC [2408] FATAL:  database "immich" does not exist
2025-10-12 11:54:05.841 UTC [2420] FATAL:  database "immich" does not exist
2025-10-12 11:54:05.860 UTC [2421] FATAL:  database "immich" does not exist
2025-10-12 11:54:10.404 UTC [2433] FATAL:  database "immich" does not exist
2025-10-12 11:54:10.461 UTC [2435] FATAL:  database "immich" does not exist
2025-10-12 11:59:10.565 UTC [2467] FATAL:  database "immich" does not exist
2025-10-12 11:59:10.609 UTC [2469] FATAL:  database "immich" does not exist
2025-10-12 12:04:10.706 UTC [2501] FATAL:  database "immich" does not exist
2025-10-12 12:04:10.749 UTC [2503] FATAL:  database "immich" does not exist
2025-10-12 12:09:10.919 UTC [2535] FATAL:  database "immich" does not exist
2025-10-12 12:09:11.002 UTC [2537] FATAL:  database "immich" does not exist
2025-10-12 12:14:11.101 UTC [2569] FATAL:  database "immich" does not exist
2025-10-12 12:14:11.144 UTC [2571] FATAL:  database "immich" does not exist
2025-10-12 12:19:11.244 UTC [2603] FATAL:  database "immich" does not exist
2025-10-12 12:19:11.290 UTC [2605] FATAL:  database "immich" does not exist
2025-10-12 12:21:43.593 UTC [2616] FATAL:  database "immich" does not exist
2025-10-12 12:21:43.611 UTC [2617] FATAL:  database "immich" does not exist
2025-10-12 12:21:44.144 UTC [2618] FATAL:  database "immich" does not exist
2025-10-12 12:21:44.163 UTC [2619] FATAL:  database "immich" does not exist
2025-10-12 12:21:45.180 UTC [2620] FATAL:  database "immich" does not exist
2025-10-12 12:21:45.189 UTC [2621] FATAL:  database "immich" does not exist
2025-10-12 12:21:45.191 UTC [2622] FATAL:  database "immich" does not exist
2025-10-12 12:21:45.207 UTC [2623] FATAL:  database "immich" does not exist
2025-10-12 12:21:45.218 UTC [2624] FATAL:  database "immich" does not exist
2025-10-12 12:21:45.224 UTC [2625] FATAL:  database "immich" does not exist
2025-10-12 12:24:11.393 UTC [2647] FATAL:  database "immich" does not exist
2025-10-12 12:24:11.437 UTC [2649] FATAL:  database "immich" does not exist
bash: line 2: pkill: command not found
bash: line 3: pkill: command not found
bash: line 4: pkill: command not found
chattr: No such file or directory while trying to stat /etc/ld.so.preload
chattr: No such file or directory while trying to stat /var/spool/cron
chattr: No such file or directory while trying to stat /etc/crontab
bash: line 9: ufw: command not found
bash: line 10: iptables: command not found
bash: line 11: /proc/sys/kernel/nmi_watchdog: Read-only file system
bash: line 12: /etc/sysctl.conf: Permission denied
bash: line 176: curl: command not found
bash: line 176: /usr/local/bin/curl: Permission denied
chmod: cannot access '/usr/local/bin/curl': No such file or directory
bash: line 177: /usr/local/bin/curl: No such file or directory
bash: line 178: /usr/local/bin/curl: No such file or directory
chmod: cannot access '/tmp/curl': No such file or directory
bash: line 181: /tmp/curl: No such file or directory
main: line 246: pkill: command not found
main: line 247: pkill: command not found
main: line 248: pkill: command not found
main: line 249: pkill: command not found
main: line 250: pkill: command not found
main: line 251: pkill: command not found
main: line 252: pkill: command not found
main: line 253: pkill: command not found
main: line 254: netstat: command not found
main: line 255: ps: command not found
main: line 256: ps: command not found
main: line 257: ps: command not found
main: line 258: pkill: command not found
main: line 259: ps: command not found
main: line 260: ps: command not found
main: line 261: ps: command not found
main: line 262: ps: command not found
main: line 263: ps: command not found
main: line 264: ps: command not found
main: line 265: pkill: command not found
main: line 266: pkill: command not found
main: line 267: ps: command not found
main: line 268: ps: command not found
main: line 269: pkill: command not found
main: line 270: netstat: command not found
main: line 271: netstat: command not found
main: line 272: netstat: command not found
main: line 273: netstat: command not found
main: line 274: netstat: command not found
main: line 275: pkill: command not found
main: line 276: pkill: command not found
main: line 277: pkill: command not found
main: line 278: pkill: command not found
main: line 279: pkill: command not found
main: line 280: pkill: command not found
main: line 281: pkill: command not found
main: line 282: pkill: command not found
main: line 283: pkill: command not found
main: line 284: pkill: command not found
main: line 285: pkill: command not found
main: line 287: pkill: command not found
main: line 288: pkill: command not found
main: line 289: pkill: command not found
main: line 290: pkill: command not found
main: line 291: pkill: command not found
main: line 292: pkill: command not found
main: line 293: pkill: command not found
main: line 294: pkill: command not found
main: line 295: pkill: command not found
main: line 296: pkill: command not found
main: line 297: pkill: command not found
main: line 298: pkill: command not found
main: line 299: pkill: command not found
main: line 300: pkill: command not found
main: line 301: pkill: command not found
main: line 302: pkill: command not found
main: line 303: ps: command not found
sed: can't read /tmp/.X11-unix/01: No such file or directory
cat: /tmp/.X11-unix/01: No such file or directory
sed: can't read /tmp/.X11-unix/11: No such file or directory
cat: /tmp/.X11-unix/11: No such file or directory
sed: can't read /tmp/.X11-unix/22: No such file or directory
cat: /tmp/.X11-unix/22: No such file or directory
sed: can't read /tmp/.systemd.1: No such file or directory
cat: /tmp/.systemd.1: No such file or directory
sed: can't read /tmp/.systemd.2: No such file or directory
cat: /tmp/.systemd.2: No such file or directory
sed: can't read /tmp/.systemd.3: No such file or directory
cat: /tmp/.systemd.3: No such file or directory
cat: /tmp/.systemd.1: No such file or directory
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
cat: /tmp/.systemd.2: No such file or directory
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
cat: /tmp/.systemd.3: No such file or directory
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
sed: can't read /tmp/.pg_stat.0: No such file or directory
cat: /tmp/.pg_stat.0: No such file or directory
sed: can't read /tmp/.pg_stat.1: No such file or directory
cat: /tmp/.pg_stat.1: No such file or directory
sed: can't read /var/lib/postgresql/data/./oka.pid: No such file or directory
cat: /var/lib/postgresql/data/./oka.pid: No such file or directory
sed: can't read /tmp/.ICE-unix/d: No such file or directory
cat: /tmp/.ICE-unix/d: No such file or directory
sed: can't read /tmp/.ICE-unix/m: No such file or directory
cat: /tmp/.ICE-unix/m: No such file or directory
main: line 329: pkill: command not found
main: line 330: pkill: command not found
main: line 331: pkill: command not found
main: line 332: pkill: command not found
main: line 333: pkill: command not found
main: line 334: pkill: command not found
main: line 335: pkill: command not found
main: line 336: ps: command not found
main: line 337: ps: command not found
main: line 338: ps: command not found
main: line 339: ps: command not found
main: line 340: ps: command not found
main: line 341: ps: command not found
/tmp/kinsing is b3039abf2ad5202f4a9363b418002351
/tmp/libsystem.so is ccef46c7edf9131ccffc47bd69eb743b
main: line 372: /etc/ld.so.preload: Permission denied
main: line 376: crontab: command not found
main: line 376: crontab: command not found
main: line 377: crontab: command not found
main: line 377: crontab: command not found
main: line 378: crontab: command not found
main: line 378: crontab: command not found
main: line 379: crontab: command not found
main: line 379: crontab: command not found
main: line 380: crontab: command not found
main: line 380: crontab: command not found
main: line 381: crontab: command not found
main: line 381: crontab: command not found
main: line 382: crontab: command not found
main: line 382: crontab: command not found
main: line 383: crontab: command not found
main: line 383: crontab: command not found
main: line 384: crontab: command not found
main: line 384: crontab: command not found
main: line 385: crontab: command not found
main: line 385: crontab: command not found
main: line 386: crontab: command not found
main: line 386: crontab: command not found
main: line 387: crontab: command not found
main: line 387: crontab: command not found
main: line 388: crontab: command not found
main: line 388: crontab: command not found
main: line 389: crontab: command not found
main: line 389: crontab: command not found
main: line 390: crontab: command not found
main: line 390: crontab: command not found
main: line 391: crontab: command not found
main: line 391: crontab: command not found
main: line 392: crontab: command not found
main: line 392: crontab: command not found
main: line 393: crontab: command not found
main: line 393: crontab: command not found
main: line 394: crontab: command not found
main: line 394: crontab: command not found
main: line 395: crontab: command not found
main: line 395: crontab: command not found
main: line 396: crontab: command not found
main: line 396: crontab: command not found
main: line 397: crontab: command not found
main: line 397: crontab: command not found
main: line 398: crontab: command not found
main: line 398: crontab: command not found
main: line 399: crontab: command not found
main: line 399: crontab: command not found
main: line 400: crontab: command not found
main: line 400: crontab: command not found
main: line 401: crontab: command not found
main: line 401: crontab: command not found
main: line 402: crontab: command not found
main: line 402: crontab: command not found
main: line 403: crontab: command not found
main: line 403: crontab: command not found
main: line 404: crontab: command not found
main: line 404: crontab: command not found
main: line 405: crontab: command not found
main: line 405: crontab: command not found
main: line 406: crontab: command not found
main: line 406: crontab: command not found
main: line 407: crontab: command not found
main: line 407: crontab: command not found
main: line 408: crontab: command not found
main: line 408: crontab: command not found
main: line 409: crontab: command not found
main: line 409: crontab: command not found
main: line 410: crontab: command not found
main: line 410: crontab: command not found
main: line 411: crontab: command not found
main: line 411: crontab: command not found
main: line 412: crontab: command not found
main: line 412: crontab: command not found
main: line 413: crontab: command not found
main: line 413: crontab: command not found
main: line 414: crontab: command not found
main: line 414: crontab: command not found
main: line 415: crontab: command not found
main: line 415: crontab: command not found
main: line 416: crontab: command not found
main: line 416: crontab: command not found
main: line 417: crontab: command not found
main: line 417: crontab: command not found
main: line 418: crontab: command not found
main: line 418: crontab: command not found
main: line 419: crontab: command not found
main: line 419: crontab: command not found
main: line 420: crontab: command not found
main: line 420: crontab: command not found
main: line 421: crontab: command not found
main: line 421: crontab: command not found
main: line 422: crontab: command not found
main: line 422: crontab: command not found
main: line 423: crontab: command not found
main: line 423: crontab: command not found
main: line 424: crontab: command not found
main: line 424: crontab: command not found
main: line 425: crontab: command not found
main: line 425: crontab: command not found
main: line 426: crontab: command not found
main: line 426: crontab: command not found
main: line 427: crontab: command not found
main: line 427: crontab: command not found
main: line 428: crontab: command not found
main: line 428: crontab: command not found
main: line 429: crontab: command not found
main: line 429: crontab: command not found
main: line 430: crontab: command not found
main: line 430: crontab: command not found
main: line 431: crontab: command not found
main: line 431: crontab: command not found
main: line 432: crontab: command not found
main: line 432: crontab: command not found
main: line 433: crontab: command not found
main: line 433: crontab: command not found
main: line 434: crontab: command not found
main: line 434: crontab: command not found
main: line 435: crontab: command not found
main: line 435: crontab: command not found
main: line 436: crontab: command not found
main: line 436: crontab: command not found
main: line 437: crontab: command not found
main: line 437: crontab: command not found
main: line 438: crontab: command not found
main: line 438: crontab: command not found
main: line 439: crontab: command not found
main: line 439: crontab: command not found
bash: line 483: crontab: command not found
bash: line 490: crontab: command not found
2025-10-12 12:29:11.540 UTC [3726] FATAL:  database "immich" does not exist
2025-10-12 12:29:11.583 UTC [3728] FATAL:  database "immich" does not exist
2025-10-12 12:34:11.687 UTC [3760] FATAL:  database "immich" does not exist
2025-10-12 12:34:11.732 UTC [3762] FATAL:  database "immich" does not exist
2025-10-12 12:39:11.843 UTC [3794] FATAL:  database "immich" does not exist
2025-10-12 12:39:11.891 UTC [3796] FATAL:  database "immich" does not exist
2025-10-12 12:44:11.991 UTC [3828] FATAL:  database "immich" does not exist
2025-10-12 12:44:12.034 UTC [3830] FATAL:  database "immich" does not exist
2025-10-12 12:45:22.939 UTC [3836] FATAL:  password authentication failed for user "postgres"
2025-10-12 12:45:22.939 UTC [3836] DETAIL:  Connection matched pg_hba.conf line 100: "host all all all scram-sha-256"
2025-10-12 12:49:12.126 UTC [3863] FATAL:  database "immich" does not exist
2025-10-12 12:49:12.173 UTC [3865] FATAL:  database "immich" does not exist
2025-10-12 12:54:05.838 UTC [3886] FATAL:  database "immich" does not exist
2025-10-12 12:54:05.859 UTC [3887] FATAL:  database "immich" does not exist
2025-10-12 12:54:12.269 UTC [3899] FATAL:  database "immich" does not exist
2025-10-12 12:54:12.318 UTC [3901] FATAL:  database "immich" does not exist
2025-10-12 12:59:12.424 UTC [3933] FATAL:  database "immich" does not exist
2025-10-12 12:59:12.472 UTC [3935] FATAL:  database "immich" does not exist
2025-10-12 13:04:12.574 UTC [3967] FATAL:  database "immich" does not exist
2025-10-12 13:04:12.618 UTC [3969] FATAL:  database "immich" does not exist
2025-10-12 13:07:10.527 UTC [3982] FATAL:  database "immich" does not exist
2025-10-12 13:07:10.534 UTC [3983] FATAL:  database "immich" does not exist
2025-10-12 13:07:10.537 UTC [3984] FATAL:  database "immich" does not exist
2025-10-12 13:07:10.550 UTC [3985] FATAL:  database "immich" does not exist
2025-10-12 13:07:10.562 UTC [3986] FATAL:  database "immich" does not exist
2025-10-12 13:07:10.567 UTC [3987] FATAL:  database "immich" does not exist
2025-10-12 13:07:10.571 UTC [3988] FATAL:  database "immich" does not exist
2025-10-12 13:07:10.578 UTC [3989] FATAL:  database "immich" does not exist
2025-10-12 13:07:10.586 UTC [3990] FATAL:  database "immich" does not exist
2025-10-12 13:07:10.590 UTC [3991] FATAL:  database "immich" does not exist
2025-10-12 13:07:10.599 UTC [3992] FATAL:  database "immich" does not exist
2025-10-12 13:07:10.607 UTC [3993] FATAL:  database "immich" does not exist
2025-10-12 13:07:11.037 UTC [3994] FATAL:  database "immich" does not exist
2025-10-12 13:07:11.057 UTC [3995] FATAL:  database "immich" does not exist
2025-10-12 13:07:13.666 UTC [3996] FATAL:  database "immich" does not exist
2025-10-12 13:07:13.683 UTC [3997] FATAL:  database "immich" does not exist
2025-10-12 13:07:13.697 UTC [3998] FATAL:  database "immich" does not exist
2025-10-12 13:07:13.705 UTC [3999] FATAL:  database "immich" does not exist
2025-10-12 13:07:15.361 UTC [4001] FATAL:  database "immich" does not exist
2025-10-12 13:07:15.363 UTC [4000] FATAL:  database "immich" does not exist
2025-10-12 13:07:15.366 UTC [4002] FATAL:  database "immich" does not exist
2025-10-12 13:07:15.382 UTC [4003] FATAL:  database "immich" does not exist
2025-10-12 13:07:15.385 UTC [4004] FATAL:  database "immich" does not exist
2025-10-12 13:07:15.394 UTC [4005] FATAL:  database "immich" does not exist
2025-10-12 13:09:12.717 UTC [4025] FATAL:  database "immich" does not exist
2025-10-12 13:09:12.759 UTC [4027] FATAL:  database "immich" does not exist
2025-10-12 13:09:26.917 UTC [4029] LOG:  invalid length of startup packet
2025-10-12 13:12:35.791 UTC [4043] FATAL:  database "immich" does not exist
2025-10-12 13:12:35.811 UTC [4044] FATAL:  database "immich" does not exist
2025-10-12 13:12:36.017 UTC [4045] FATAL:  database "immich" does not exist
2025-10-12 13:12:36.030 UTC [4046] FATAL:  database "immich" does not exist
2025-10-12 13:12:36.042 UTC [4047] FATAL:  database "immich" does not exist
2025-10-12 13:12:36.050 UTC [4048] FATAL:  database "immich" does not exist
2025-10-12 13:14:12.866 UTC [4066] FATAL:  database "immich" does not exist
2025-10-12 13:14:12.911 UTC [4068] FATAL:  database "immich" does not exist

Additional information

No response

[dragonfly8677]

I've just restored from backup and am facing the same issue. I assume you didn't solve it? I'm considering doing a fresh install, it would just be a pain to lose all my albums.

[lorenzo-morelli]

I actually managed to solve it, though my root cause was a bit of a nightmare: a malware called Kinsing. It was repeatedly killing my Postgres container to hijack the server resources for Bitcoin mining. Once I purged the malicious processes and tightened my security, everything started working normally. If you aren't dealing with a security breach but the restoration is still failing, I'd recommend a fresh install of Immich, followed by a database restoration from your backup. P.S.: As of v2.5.0, Immich actually allows you to restore the database directly from the Web UI startup screen, which might save you a lot of manual configuration hassle!