Immich v2.4.1 → v2.5.0 Manual Upgrade Failures

[microsoftShannon]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

Environment

Proxmox Setup:

• Proxmox VE 8.x
• LXC Container (Unprivileged)
• Container ID:
• OS: Debian 13 (Trixie)
• CPU: 4 cores
• RAM: 4GB (later increased to 8GB)
• Storage: 20GB on local SSD

Immich Installation:

• Installed via: Community Scripts ct/immich.sh
• Original Version: v2.4.1
• Installation Type: Native (not Docker)
• Services: systemd (immich-web, immich-ml)
• Database: PostgreSQL 16
• Storage: NFS mount (bind-mounted from Proxmox host)

Network:

• IP: x.x.x.x (DHCP reservation)
• Access: http://x.x.x.x:2283

Issue Description

Attempted manual upgrade from v2.4.1 to v2.5.0 following standard procedure, but encountered multiple cascading failures related to module resolution, Python dependencies, and file structure issues.

Attempted Upgrade Process

Step 1: Pre-Upgrade Backup ✅

# Created PostgreSQL backup
su - postgres -c "pg_dump immich -F c -f /var/lib/postgresql/backups/immich_pre_upgrade_$(date +%Y%m%d).backup"
# Result: 238MB backup created successfully

Step 2: Download & Extract v2.5.0 Source

cd /opt/immich
systemctl stop immich-web immich-ml
rm -rf source
wget https://github.com/immich-app/immich/archive/refs/tags/v2.5.0.tar.gz
tar -xzf v2.5.0.tar.gz
mv immich-2.5.0 source
rm v2.5.0.tar.gz

Step 3: Rebuild Server ❌

cd /opt/immich/source/server
pnpm install  # Completed successfully
pnpm build    # Completed successfully

# Copied built files to app directory
rm -rf /opt/immich/app/*
cp -r dist/* /opt/immich/app/
cp package.json /opt/immich/app/
cp -r node_modules /opt/immich/app/

Problem 1: Wrong directory structure - Files were flattened instead of preserving dist/ folder hierarchy.

Errors Encountered

Error 1: Module Not Found (kysely)

Symptom:

Error: Cannot find module 'kysely'
Require stack:
- /opt/immich/app/dist/main.js

Root Cause:

• Copied dist/* instead of dist/ directory
• Node modules symlinks pointing to wrong paths
• pnpm workspace structure not preserved

Attempted Fixes:

1. ✅ Moved files into proper dist/ structure
2. ✅ Reinstalled node_modules: pnpm install --prod
3. ❌ Symlinks still broken: /opt/immich/app/node_modules/kysely -> .pnpm/kysely@0.28.2/node_modules/kysely
4. ❌ Target path didn't exist properly

Error 2: Python Dependencies Missing

Symptom:

# ML service failing
ModuleNotFoundError: No module named 'gunicorn'

Root Cause:

• Machine learning Python dependencies not installed
• No pip available (Python 3.13 minimal install)

Attempted Fixes:

1. ❌ pip install --break-system-packages -e . - pip not found
2. ❌ pip3 install --break-system-packages -e . - pip3 not found
3. ✅ uv pip install --system -e . - Worked, but...

Error 3: Python Executable Not Found

Symptom:

FileNotFoundError: [Errno 2] No such file or directory: 'python'

Root Cause:

• System has python3 but not python
• ML startup script expects python command

Fix Applied:

ln -s /usr/bin/python3 /usr/bin/python

Error 4: Module Resolution Persists

Even after multiple attempts to fix node_modules:

• Reinstalling in /opt/immich/app
• Copying from /opt/immich/source/server
• Fixing ownership (chown -R immich:immich)

Consistent Error:

Error: Cannot find module 'kysely'

The pnpm symlink structure worked when running as root but failed when running as the immich user via systemd.

Database Restoration Issues

When attempting to restore from backup after failed upgrade:

Error:

ERROR: insert or update on table "album" violates foreign key constraint
ERROR: duplicate key value violates unique constraint

Cause: Fresh v2.4.1 installation had pre-created schema that conflicted with backup.

Solution:

su - postgres -c "dropdb immich"
su - postgres -c "createdb immich -O immich"
gunzip -c backup.sql.gz | su - postgres -c "psql immich"

Resolution

Abandoned manual upgrade and performed clean rebuild:

1. Destroyed broken container: pct destroy xxx
2. Rebuilt using community script: Fresh v2.4.1 installation
3. Restored database backup: All 54,542 photos recovered
4. Reconnected NFS storage: Bind-mounted Waihona
5. Fixed database password mismatch:

   su - postgres -c "psql -c \"ALTER USER immich WITH PASSWORD 'PASSWORD_FROM_ENV';\""

Total time: ~3 hours of troubleshooting, 30 minutes for clean rebuild.

Recommendations

For Community Script Maintainers

1. Add proper update mechanism to ct/immich.sh:

   function update_script() {
     # Check version
     # Stop services
     # Backup database automatically
     # Download new version
     # Rebuild with proper structure
     # Migrate if needed
     # Restart services
   }

2. Document upgrade path clearly:

   • Which versions support in-place upgrades
   • When clean reinstall is recommended
   • How to preserve data during upgrades

3. Add version-specific migration scripts for major version changes

4. Ensure Python dependencies are installed with correct package manager (uv)

5. Create python symlink during initial installation

6. Test pnpm workspace structure in unprivileged containers

For Users

DO NOT attempt manual upgrades. Wait for official community script support:

# When officially supported, use:
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/immich.sh)"
# And select "Update" option (when available)

Before any upgrade attempt:

1. ✅ Create Proxmox container backup
2. ✅ Verify database backups exist (daily at 2 AM)
3. ✅ Document current configuration
4. ✅ Test backup restoration

Technical Details

File Structure Issues

Expected:

/opt/immich/app/
├── dist/
│   ├── main.js
│   ├── *.js
│   └── [subdirectories]
├── node_modules/
├── package.json
└── machine-learning/
    └── ml_start.sh

What happened during manual upgrade:

/opt/immich/app/
├── main.js        # ❌ Flattened!
├── *.js           # ❌ All in root
├── node_modules/
│   └── kysely -> ../../node_modules/.pnpm/...  # ❌ Broken symlink
└── package.json

Working Configuration (v2.4.1)

Database Backup Schedule:

"backup": {
  "database": {
    "cronExpression": "0 02 * * *",
    "enabled": true,
    "keepLastAmount": 14
  }
}

Storage:

• Original photos: /mnt/xxx/library/ (never modified)
• Thumbnails: /mnt/xxx/thumbs/
• Backups: /mnt/xxx/backups/

Related Issues

• pnpm workspace symlinks in unprivileged LXC containers
• Python package management in Debian 13 (minimal install)
• Module resolution when running as non-root user

System Information

# Container
root@immich:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
VERSION="13 (trixie)"

# Node.js
root@immich:~# node --version
v24.13.0

# pnpm
root@immich:~# pnpm --version
10.28.0

# Python
root@immich:~# python3 --version
Python 3.13.5

# PostgreSQL
root@immich:~# su - postgres -c "psql --version"
psql (PostgreSQL) 16.11

Conclusion

Manual upgrades of Immich in the community-scripts environment are not recommended due to:

• Complex pnpm workspace structure
• Python dependency management challenges
• Service file structure expectations
• Database migration requirements

Recommendation: Wait for official community script update support for v2.5.0 before upgrading.

---

Labels: bug, enhancement, documentation, immich, upgrade
Milestone: Future release with update support

The OS that Immich Server is running on

Debian 13 (Trixie)

Version of Immich Server

2.5.0

Version of Immich Mobile App

2.4.1 build .239

Platform with the issue

• Server
• Web
• Mobile

Device make and model

Iphone 16 Pro Max

Your docker-compose.yml content

N/A - Not using Docker

Installation Method: Native systemd deployment via Proxmox Community Scripts
Repository: https://github.com/community-scripts/ProxmoxVE
Script: ct/immich.sh
Deployment: LXC container with systemd services

See systemd service configuration above for equivalent deployment details.

Your .env content

bash# Database Configuration
DB_HOSTNAME=127.0.0.1
DB_USERNAME=immich
DB_PASSWORD=********  # REDACTED
DB_DATABASE_NAME=immich
DB_VECTOR_EXTENSION=vectorchord

# Storage Configuration
IMMICH_MEDIA_LOCATION=/mnt/waihona

Reproduction steps

Reproduction Steps for Immich v2.4.1 → v2.5.0 Upgrade Failure
markdown## Prerequisites

Environment

• Proxmox VE 8.x
• LXC Container (Unprivileged)
• OS: Debian GNU/Linux 13 (Trixie)
• Architecture: x86_64

Initial Installation

# Install Immich v2.4.1 using Proxmox Community Scripts
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/immich.sh)"

Container Configuration:

• Type: Unprivileged
• CPU: 4 cores
• RAM: 4096 MB
• Disk: 20 GB
• Network: DHCP (example: 192.168.x.x)

Verify Working Installation

# Access web interface
http://:2283

# Verify services running
systemctl status immich-web immich-ml
# Both should show: active (running)

# Verify version
cat /opt/immich/app/package.json | grep version
# Should show: "version": "2.4.1"

---

Reproduction Steps

Step 1: Create Database Backup

# Enter container (from Proxmox host)
pct enter 

# Create backup directory
mkdir -p /var/lib/postgresql/backups
chown postgres:postgres /var/lib/postgresql/backups

# Create PostgreSQL backup
su - postgres -c "pg_dump immich -F c -f /var/lib/postgresql/backups/immich_pre_upgrade_$(date +%Y%m%d).backup"

# Verify backup created
ls -lh /var/lib/postgresql/backups/
# Expected: Backup file ~200-300MB

Step 2: Stop Immich Services

systemctl stop immich-web
systemctl stop immich-ml

# Verify stopped
systemctl status immich-web immich-ml
# Both should show: inactive (dead)

Step 3: Download v2.5.0 Source

cd /opt/immich

# Remove old source
rm -rf source

# Download v2.5.0
wget https://github.com/immich-app/immich/archive/refs/tags/v2.5.0.tar.gz
tar -xzf v2.5.0.tar.gz
mv immich-2.5.0 source
rm v2.5.0.tar.gz

# Verify download
ls -la /opt/immich/source/

Step 4: Build Server Application

cd /opt/immich/source/server

# Install dependencies
pnpm install

Expected Output:

Packages: +2553
Progress: resolved 2706, reused 2135, downloaded 410, added 2553, done
Done in 38.9s using pnpm v10.28.0

# Build application
pnpm build

Expected Output:

> immich@2.5.0 build /opt/immich/source/server
> nest build

Step 5: Copy Built Files to App Directory

⚠️ This step demonstrates the incorrect method that causes the bug:

cd /opt/immich

# Clear app directory
rm -rf app/*

# Copy files (WRONG - flattens directory structure)
cp -r source/server/dist/* app/
cp source/server/package.json app/
cp -r source/server/node_modules app/

Result: Files are incorrectly flattened:

/opt/immich/app/
├── main.js              # ❌ Should be in dist/
├── app.module.js        # ❌ Should be in dist/
├── controllers/         # ❌ Should be in dist/
├── node_modules/        # ✓ Correct location
└── package.json         # ✓ Correct location

Expected structure:

/opt/immich/app/
├── dist/                # ✓ Subdirectory should exist
│   ├── main.js
│   ├── app.module.js
│   └── controllers/
├── node_modules/
└── package.json

Step 6: Attempt to Start Web Service

systemctl start immich-web

# Check status
systemctl status immich-web

Expected Error:

× immich-web.service - immich Web Service
     Active: failed (Result: exit-code)

Check logs:

tail -30 /var/log/immich/web.log

Expected Error in Logs:

Error: Cannot find module 'kysely'
Require stack:
- /opt/immich/app/dist/main.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1421:15)
    ...
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/opt/immich/app/dist/main.js' ]

---

Attempted Fixes (All Failed)

Fix Attempt #1: Correct Directory Structure

cd /opt/immich/app

# Create dist directory
mkdir -p dist

# Move files into dist/
mv *.js *.map *.d.ts *.tsbuildinfo dist/
mv bin commands controllers cores dtos emails maintenance middleware \
   repositories schema services sql-tools types utils workers dist/

# Verify structure
ls -la /opt/immich/app/dist/main.js
# Should exist now

Restart service:

systemctl restart immich-web
tail -20 /var/log/immich/web.log

Result: Same MODULE_NOT_FOUND error persists

---

Fix Attempt #2: Reinstall node_modules

cd /opt/immich/app

# Remove existing node_modules
rm -rf node_modules

# Reinstall dependencies
pnpm install --prod

Expected Output:

Packages: +636
Done in 8.1s using pnpm v10.28.0

Check symlink structure:

ls -la /opt/immich/app/node_modules/kysely

Output:

lrwxrwxrwx 1 root root 39 -> .pnpm/kysely@0.28.2/node_modules/kysely

Verify target exists:

ls -la /opt/immich/app/node_modules/.pnpm/kysely@0.28.2/node_modules/kysely

Output:

drwxr-xr-x 4 root root 4096 dist/
-rw-r--r-- 3 1082 package.json
...

Restart service:

systemctl restart immich-web

Result: Still fails with MODULE_NOT_FOUND

---

Fix Attempt #3: Test Module Resolution

# Test as root user
cd /opt/immich/app
node -e "console.log(require('kysely'))"

Output:

{
  Kysely: [Getter],
  sql: [Getter],
  ...
}

✅ Works as root!

But service running as immich user still fails:

systemctl start immich-web
tail /var/log/immich/web.log

Result: MODULE_NOT_FOUND when running as immich user

---

Fix Attempt #4: Fix Ownership

# Set proper ownership
chown -R immich:immich /opt/immich/app

# Restart service
systemctl restart immich-web

Result: Still fails with same error

---

Machine Learning Service Issues

Issue #1: Python Dependencies Missing

systemctl start immich-ml
systemctl status immich-ml

Expected Error:

× immich-ml.service - immich Machine-Learning
     Active: failed (Result: exit-code)

Check logs:

tail /var/log/immich/ml.log

Expected Error:

ModuleNotFoundError: No module named 'gunicorn'

Attempt to install:

cd /opt/immich/source/machine-learning

pip install --break-system-packages -e .

Result:

bash: pip: command not found

pip3 install --break-system-packages -e .

Result:

bash: pip3: command not found

python3 -m pip install --break-system-packages -e .

Result:

/bin/python3: No module named pip

---

Issue #2: Install with uv (Partial Success)

# Check if uv is available
which uv

Output:

/usr/local/bin/uv

Install Python dependencies:

cd /opt/immich/source/machine-learning
uv pip install --system -e .

Expected Output:

Successfully installed [packages]

Try to start ML service:

systemctl start immich-ml
tail /var/log/immich/ml.log

Expected Error:

FileNotFoundError: [Errno 2] No such file or directory: 'python'

---

Issue #3: Python Executable Not Found

Fix:

# Create python symlink
ln -s /usr/bin/python3 /usr/bin/python

# Verify
which python
python --version

Output:

/usr/bin/python
Python 3.13.5

Start ML service:

systemctl start immich-ml
systemctl status immich-ml

Result: ✅ ML service now starts successfully

● immich-ml.service - immich Machine-Learning
     Active: active (running)

---

Final State

After all attempted fixes:

• ✅ ML Service: Running (after Python fixes)
• ❌ Web Service: Failed (MODULE_NOT_FOUND persists)
• ❌ Database: Not migrated
• ❌ Photos: Inaccessible via web UI

---

System Information

# Container OS
cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
VERSION="13 (trixie)"
VERSION_ID="13"

# Node.js version
node --version

v24.13.0

# pnpm version
pnpm --version

10.28.0

# Python version
python3 --version

Python 3.13.5

# PostgreSQL version
su - postgres -c "psql --version"

psql (PostgreSQL) 16.11

---

Working Restoration Method

After failed upgrade attempts, complete restoration was achieved:

Step 1: Destroy Broken Container

# From Proxmox host
pct stop 
pct destroy 

Step 2: Reinstall Fresh v2.4.1

# Use community script
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/immich.sh)"

# Use same container ID and configuration as before

Step 3: Reconnect NFS Storage

# On Proxmox host
mount -t nfs :/volume1/storage /mnt/storage

# Bind mount to container
pct stop 
pct set  -mp0 /mnt/storage,mp=/mnt/storage
pct start 

Step 4: Restore Database

# Enter container
pct enter 

# Drop and recreate database
su - postgres -c "dropdb immich"
su - postgres -c "createdb immich -O immich"

# Restore from backup
gunzip -c /mnt/storage/backups/immich_pre_upgrade_20260127.backup.sql.gz | \
  su - postgres -c "psql immich"

# Verify restoration
su - postgres -c "psql -d immich -c 'SELECT COUNT(*) FROM asset;'"
# Should show your photo count

Step 5: Fix Database Password

# Get password from .env
grep DB_PASSWORD /opt/immich/.env

# Set password in PostgreSQL
su - postgres -c "psql -c \"ALTER USER immich WITH PASSWORD '';\""

Step 6: Configure Storage Location

# Add storage location to .env
echo "IMMICH_MEDIA_LOCATION=/mnt/storage" >> /opt/immich/.env

Step 7: Start Services

systemctl start immich-ml
systemctl start immich-web

# Verify both running
systemctl status immich-web immich-ml

Result: ✅ Full restoration on v2.4.1 with all photos accessible

---

Root Causes Identified

1. pnpm Workspace Symlinks

   • Symlinks work for root user but fail for non-root systemd services
   • Unprivileged LXC container permission issues
   • Module resolution path incorrect when files copied vs installed

2. Python Package Management

   • Debian 13 minimal install lacks pip/pip3
   • Only uv package manager available
   • Python symlink not created by default

3. No Migration Path

   • Community script designed for fresh installs only
   • No documented upgrade procedure
   • No automated database migration

4. File Structure Assumptions

   • Systemd service expects specific directory layout
   • Simple file copy breaks pnpm workspace structure
   • Build artifacts require precise placement

---

Expected Behavior

The community script should provide:

1. Update mechanism:

   bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/immich.sh)"
   # Should detect existing installation and offer update option

2. Automatic database backup before upgrade

3. Proper file structure preservation during upgrade

4. Database migration handled automatically

5. Service restart with health checks

6. Rollback capability if upgrade fails

---

Workaround for Now

Do NOT attempt manual upgrade. Wait for official community script support for v2.5.0.

To stay safe:

• Remain on v2.4.1
• Ensure daily database backups are running
• Create Proxmox container backups regularly
• Monitor https://github.com/community-scripts/ProxmoxVE for updates

---

Additional Notes

• Total troubleshooting time: ~3 hours
• Clean rebuild and restoration time: ~30 minutes
• No photo data lost (stored on NFS, independent of container)
• Database successfully restored from automatic backup

Relevant log output

Additional information

No response

[bo0tzz]

You're using an unsupported installation method, so there's no point in reporting this to us as we can't help with it.

[Mraedis]

For the future, the proper way to update the LXC script is to wait for the maintainers to release the update and then from the commandline just... update .

[MickLesk]

Damn it, we're pinning this version specifically because there are such big changes, and users are doing user stuff...

[tremor021]

Please don't post issues with Immich on this issues board if you're using our install script.
Keep all bugs related to Immich when using our script to our own github. We are carefully testing every version bump to ensure Immich runs smoothly on our baremetal install.

Thank you