Bootstrap Failure Recovery

Use this runbook when first-boot VPS init does not finish successfully. The sequence is explicit and safe to execute end-to-end.

0) Preparation (do not skip)

Keep provider web console access open (rescue path if SSH is broken).
Run recovery as root (provider console) or as DEVOPS_USER / COOLIFY_SUDO_NOPASSWD_USER (passwordless sudo).
Keep local source of truth ready:
- local repo: public-vps-coolify-bootstrap
- local env file: bootstrap-artifacts/bootstrap.env
Set shell variables on your machine for faster copy/paste:

export SERVER_IP="<server-ip>"
export SSH_PORT="<ssh-port>"
export DEVOPS_USER="<devops-user>"

1) Confirm first-boot init failure state

sudo cloud-init status --wait
sudo cloud-init status --long
sudo cloud-init query --all | head -n 40

Note: cloud-init is the native Ubuntu first-boot service provided by the base image. This repository does not create that service; bootstrap scripts run inside its first-boot execution.

Where to run these commands:

if SSH is not reachable yet, run them in provider web console (serial/VNC console)
example (Hetzner Cloud): Servers -> <server> -> Console
run by SSH only after SSH access is confirmed

Ready for SSH (quick 3-command checklist, run on server):

sudo cloud-init status --wait
sudo ss -lntp | grep -E ':(<SSH_PORT>)\b' || true
sudo ufw status verbose

If status is done, the first-boot init service finished, and you should troubleshoot service-level issues instead. If status is error or still not complete, continue.

2) Collect first error evidence

sudo tail -n 250 /var/log/vps-bootstrap.log
sudo journalctl -u cloud-init -u cloud-config -u cloud-final --no-pager -n 250
sudo grep -Rni "error\\|failed\\|traceback" /var/log/cloud-init* 2>/dev/null | tail -n 80

/var/log/vps-bootstrap.log is the primary consolidated bootstrap log and includes timestamped, script-context lines prefixed with [SUCCESS], [WARNING], [ERROR].

Typical root causes:

unresolved CHANGE_ME values in generated VPS-Coolify init file
invalid SSH_PUBLIC_KEY format
clone failure for BOOTSTRAP_REPO_URL / BOOTSTRAP_REPO_REF
transient apt/network failures during bootstrap

2.1) Emergency SSH recovery from provider console

If SSH is unreachable, run this directly in provider console (root shell):

sudo bash /opt/vps-coolify-bootstrap/scripts/recover-ssh-access.sh

What it does in default mode:

creates /etc/ssh/sshd_config.d/10-port-recovery.conf with temporary dual-port listen (22 + configured SSH_PORT)
disables ssh.socket, enables/restarts ssh.service, validates sshd -t
opens UFW rules for 22 and configured SSH_PORT

Optional:

# if Fail2ban blocked your source IP
sudo bash /opt/vps-coolify-bootstrap/scripts/recover-ssh-access.sh --unban-ip <your_public_ip>

# after SSH on configured port works, close temporary port 22 again
sudo bash /opt/vps-coolify-bootstrap/scripts/recover-ssh-access.sh --close-22

3) Validate minimum host baseline

# Extract DEVOPS/COOLIFY users from server-side bootstrap env.
devops_user="$(sudo sed -n 's/^DEVOPS_USER=//p' /etc/vps-coolify-bootstrap/bootstrap.env | tr -d \"'\\r\")"
coolify_user="$(sudo sed -n 's/^COOLIFY_SUDO_NOPASSWD_USER=//p' /etc/vps-coolify-bootstrap/bootstrap.env | tr -d \"'\\r\")"
devops_user="${devops_user:-devops}"
coolify_user="${coolify_user:-coolify}"
id "$devops_user" || true
id "$coolify_user" || true
command -v docker || true
sudo systemctl is-active ssh.service fail2ban unattended-upgrades || true
sudo ufw status verbose || true

If core services are missing or inactive, continue with manual bootstrap replay.

4) Ensure bootstrap repo exists on server

Check if the first-boot init stage cloned the bootstrap repo:

sudo test -d /opt/vps-coolify-bootstrap && echo "repo present" || echo "repo missing"

If missing, recreate it with the same ref used by your env:

sudo rm -rf /opt/vps-coolify-bootstrap
sudo git clone --depth 1 --branch <BOOTSTRAP_REPO_REF> <BOOTSTRAP_REPO_URL> /opt/vps-coolify-bootstrap

Validate script path:

sudo test -f /opt/vps-coolify-bootstrap/scripts/bootstrap-host.sh && echo "script ok"

5) Ensure server env file exists and is correct

Required server env path:

/etc/vps-coolify-bootstrap/bootstrap.env

If missing or incorrect, recreate it from your local bootstrap-artifacts/bootstrap.env values. Then verify required keys on server:

sudo awk -F= '
/^(SSH_PORT|DEVOPS_USER|COOLIFY_SUDO_NOPASSWD_USER|ADDITIONAL_SUDO_USERS|SSH_PUBLIC_KEY|COOLIFY_PUBLIC_DOMAIN|CLOSE_COOLIFY_REALTIME_PORTS|DOCKER_DISABLE_IPV6_FOR_PARSEADDR_FIX|COOLIFY_REALTIME_DOMAIN|COOLIFY_ROOT_USERNAME|COOLIFY_ROOT_USER_EMAIL|COOLIFY_ROOT_USER_PASSWORD|USER_PASSWORDS_ENCRYPTION_PASSWORD)=/ {
  print $1"=<set>"
}
' /etc/vps-coolify-bootstrap/bootstrap.env

Hard checks before replay:

if sudo grep -n "CHANGE_ME" /etc/vps-coolify-bootstrap/bootstrap.env; then
  echo "replace placeholders first"
fi
sudo sed -n 's/^COOLIFY_ROOT_USER_PASSWORD=//p' /etc/vps-coolify-bootstrap/bootstrap.env | tr -d "'\r" | awk '{print length($0)}'
sudo sed -n 's/^USER_PASSWORDS_ENCRYPTION_PASSWORD=//p' /etc/vps-coolify-bootstrap/bootstrap.env | tr -d "'\r" | awk '{print length($0)}'
root_pw="$(sudo sed -n 's/^COOLIFY_ROOT_USER_PASSWORD=//p' /etc/vps-coolify-bootstrap/bootstrap.env | tr -d "'\r")"
if [[ ${#root_pw} -ge 16 && "$root_pw" =~ [a-z] && "$root_pw" =~ [A-Z] && "$root_pw" =~ [0-9] && "$root_pw" =~ [^[:alnum:]] ]]; then
  echo "COOLIFY_ROOT_USER_PASSWORD complexity: OK"
else
  echo "COOLIFY_ROOT_USER_PASSWORD complexity: INVALID"
fi

COOLIFY_ROOT_USER_PASSWORD must be at least 16 characters and include lowercase, uppercase, digit, and symbol. USER_PASSWORDS_ENCRYPTION_PASSWORD must be at least 16 characters.

6) Replay bootstrap manually (authoritative recovery step)

Run:

sudo bash /opt/vps-coolify-bootstrap/scripts/bootstrap-host.sh /etc/vps-coolify-bootstrap/bootstrap.env

This script is idempotent and executes the following actions in order:

create/repair users and SSH keys (including COOLIFY_SUDO_NOPASSWD_USER)
during this on-host replay, set passwords for managed users (DEVOPS_USER, COOLIFY_SUDO_NOPASSWD_USER, ADDITIONAL_SUDO_USERS) only when account is locked/unset or missing from vault (not a local pre-generation step)
store generated credentials encrypted in /etc/vps-coolify-bootstrap/user-passwords.enc
sync AllowUsers from effective managed users
enforce SSH runtime/config checks
terminate stale sshd listeners on 22 when SSH_PORT is not 22
reset/apply UFW baseline rules (SSH_PORT, 80, 443)
enable fail2ban and unattended-upgrades
install/start Coolify if missing
apply Docker ParseAddr workaround policy when enabled (DOCKER_DISABLE_IPV6_FOR_PARSEADDR_FIX=true)
ensure Coolify root account exists (RootUserSeeder + DB verification)
sync Coolify localhost server connection to COOLIFY_SUDO_NOPASSWD_USER + SSH_PORT and dedicated localhost SSH key
sync realtime host env (PUSHER_HOST, PUSHER_PORT, PUSHER_SCHEME) from effective realtime domain (COOLIFY_REALTIME_DOMAIN or COOLIFY_PUBLIC_DOMAIN fallback)
enforce sudo/docker/coolify memberships and sudo policy (passwordless for DEVOPS_USER and COOLIFY_SUDO_NOPASSWD_USER by default)
sync DOCKER-USER guards:
- block public 6001/6002 when CLOSE_COOLIFY_REALTIME_PORTS=true

Important: Docker-published ports can bypass UFW rules. Validate exposed ports after recovery with ss -tulpen and docker ps --format 'table \t'.

7) Validate recovery on server

sudo systemctl is-active ssh.service fail2ban unattended-upgrades
sudo ufw status verbose
sudo docker ps --format 'table \t'
sudo iptables -S DOCKER-USER | grep -E '6001|6002' || true
sudo ip6tables -S DOCKER-USER 2>/dev/null | grep -E '6001|6002' || true
devops_user="$(sudo sed -n 's/^DEVOPS_USER=//p' /etc/vps-coolify-bootstrap/bootstrap.env | tr -d \"'\\r\")"
devops_user="${devops_user:-devops}"
id "$devops_user" || true
getent group sudo docker coolify
sudo bash /opt/vps-coolify-bootstrap/scripts/verify-bootstrap-state.sh /etc/vps-coolify-bootstrap/bootstrap.env

Expected:

SSH service active (not socket-activated)
UFW enabled and allowing configured SSH port + 80/443
Docker available
Coolify container running (name usually coolify)
Coolify localhost server (id 0) uses COOLIFY_SUDO_NOPASSWD_USER and configured SSH_PORT
DOCKER-USER rules present for 6001/6002 when CLOSE_COOLIFY_REALTIME_PORTS=true

8) Validate remote access from your machine

ssh -p "$SSH_PORT" "$DEVOPS_USER@$SERVER_IP" "whoami && hostname && id"

If login fails but provider console works, re-check:

SSH_PORT value in server env
firewall rule for that port (ufw status)
user presence in /etc/passwd and authorized key content
REMOTE HOST IDENTIFICATION HAS CHANGED warning (Windows/Linux/macOS): verify host fingerprint in provider console (sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub), then run ssh-keygen -R "[$SERVER_IP]:$SSH_PORT" locally and reconnect. If SSH still reports an offending key line, remove that line from local known_hosts (C:\Users\<you>\.ssh\known_hosts on Windows, ~/.ssh/known_hosts on Linux/macOS), then connect again and accept the new host key.

8A) Recover Coolify “Server is not reachable” onboarding errors

Use this section when Coolify UI shows:

ssh: connect to host ... port ...: Connection refused
sudo: a terminal is required ... / sudo: a password is required
ParseAddr(".../64"): unexpected character

A) Validate host/port/user used by Coolify

For a local server managed by the same Coolify instance:

Host: host.docker.internal
Port: SSH_PORT from bootstrap.env
User: COOLIFY_SUDO_NOPASSWD_USER (recommended, default coolify) or another user with explicit passwordless sudo policy

Important: bootstrap replay already syncs the local Coolify server to COOLIFY_SUDO_NOPASSWD_USER. If UI still shows root, run Step 6 again.

Do not use 127.0.0.1 for this case.

sudo docker exec coolify getent hosts host.docker.internal
sudo ss -lntp | grep sshd

B) Clear fail2ban/UFW blocks against Docker bridge source IPs

sudo fail2ban-client status sshd
sudo ufw status numbered

If you see a banned Docker bridge IP (for example 10.0.1.5), unban it and remove the related UFW reject rule:

sudo fail2ban-client set sshd unbanip 10.0.1.5
sudo ufw delete <rule-number-for-reject>
sudo ufw insert 1 allow from 10.0.0.0/8 to any port "$SSH_PORT" proto tcp
sudo ufw reload

Optional persistent fail2ban allow-list for private bridge ranges:

sudo tee /etc/fail2ban/jail.d/20-local-ignore-docker.conf >/dev/null <<'EOF'
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
EOF
sudo systemctl restart fail2ban

C) Fix sudo policy for Coolify-managed SSH user

Coolify server validation runs non-interactively and needs passwordless sudo. By default, DEVOPS_USER and COOLIFY_SUDO_NOPASSWD_USER are passwordless.

Check effective sudo mode:

sudo -u <coolify-ssh-user> -H bash -lc 'sudo -n true && echo OK_NOPASSWD'

If this fails, set policy permanently by updating COOLIFY_SUDO_NOPASSWD_USER in server bootstrap.env and replaying bootstrap, or add an explicit override file loaded after bootstrap policy:

sudo tee /etc/sudoers.d/zz-coolify-nopasswd >/dev/null <<'EOF'
coolify ALL=(ALL:ALL) NOPASSWD:ALL
EOF
sudo chmod 440 /etc/sudoers.d/zz-coolify-nopasswd
sudo visudo -c

D) Fix `ParseAddr(".../64")` proxy start failures

If logs show ParseAddr(".../64"), validate both layers:

Coolify host/IP field format:
- valid host example: 2001:db8:1c1c:ad5f::1
- invalid host format: 2001:db8:1c1c:ad5f::1/64
Docker IPv6 gateway parsing path (known upstream issue in vulnerable Docker behavior)

Bootstrap mitigation:

keep DOCKER_DISABLE_IPV6_FOR_PARSEADDR_FIX=true in /etc/vps-coolify-bootstrap/bootstrap.env (default)
replay bootstrap:

sudo bash /opt/vps-coolify-bootstrap/scripts/bootstrap-host.sh /etc/vps-coolify-bootstrap/bootstrap.env

Quick checks:

docker version
sudo docker info --format ''

Then retry Start Proxy in Coolify UI.

9) If replay still fails, capture focused diagnostics

sudo journalctl -xe --no-pager -n 200
sudo tail -n 200 /var/log/auth.log
sudo tail -n 200 /var/log/ufw.log 2>/dev/null || true
sudo docker logs coolify --tail 200 2>/dev/null || true

Fix the specific failing cause, then rerun Step 6.

10) Clean rebuild path (last resort)

Use this only if server state is inconsistent or unrecoverable.

PowerShell note for this section:

Prefer pwsh -File ...
If pwsh is unavailable, run the same script with powershell -ExecutionPolicy Bypass -File ...
Optional install (pwsh): winget install --id Microsoft.PowerShell --source winget
If winget is unavailable: https://github.com/PowerShell/PowerShell/releases/latest

On local machine, reset to clean env baseline:

bash scripts/generate-secrets.sh

Windows PowerShell:

pwsh -File scripts/generate-secrets.ps1

Fill all required values in bootstrap-artifacts/bootstrap.env (no CHANGE_ME).

Regenerate VPS-Coolify init:

bash scripts/prepare-vps-coolify-init.sh --overwrite