Recover from a Compromised Server

A compromised server is a bad afternoon, not the end of the world, provided you act in the right order: contain, preserve, investigate, rebuild, then harden. This is the checklist our engineers would walk you through, written down so you can start before the ticket is answered.

Signs it has happened#

• Sustained outbound traffic or CPU you did not cause (see the portal graphs).
• An abuse report from us naming your IP; unknown cron jobs, users, or SSH keys; processes with odd names; a defaced site; your mail suddenly bouncing.

1. Contain#

1. Take a snapshot in the cloud portal before touching anything. It preserves evidence and gives you something to look at later without the attacker still moving. See Snapshots.
2. Cut its reach. From the VNC console (so you are not relying on possibly-tampered SSH), block everything except your own IP: ufw default deny incoming; ufw default deny outgoing; ufw allow from YOUR.IP; ufw enable. If it is actively attacking others, power it off from the portal instead and work from rescue mode.
3. Assume every secret on the box is burned. Rotate them now: cloud portal and MyNobull passwords, SSH keys, database passwords, API tokens, application admin accounts, anything in .env files.

2. Investigate (briefly)#

You are trying to answer two questions: how did they get in, and what did they touch. Spend an hour, not a week.

last -a | head -50                       # logins and their sources
journalctl -u ssh --since "-14d" | grep -i accepted
cat /etc/passwd | tail; getent group sudo wheel
find / -xdev -newermt "$(date -d '-7 days' +%F)" -type f 2>/dev/null | grep -vE '^/(proc|sys|var/log|var/cache)' | head -100
crontab -l; ls -la /etc/cron* /var/spool/cron
for u in root $(ls /home); do echo "== $u"; cat /home/$u/.ssh/authorized_keys /root/.ssh/authorized_keys 2>/dev/null; done
ss -tunap | grep -v 127.0.0.1               # what is talking to whom
ps auxf | less                              # anything you do not recognise

On a web server, also check the site's own logs for POSTs to unexpected files (web shells), and the CMS's plugin list against what you installed. Note what you find; it goes in the ticket and it drives step 4.

3. Rebuild, do not clean#

Cleaning a compromised system is a bet that you found everything. Rebuild instead: reinstall from a fresh image, then restore data only from a backup that predates the compromise (or from the snapshot, mounted read-only in rescue mode and copied selectively). Reinstall applications from their packages, not from the old disk, and re-enter secrets fresh. If you restore a database, inspect it for injected admin users or content.

4. Close the door they used#

• Password SSH login: switch to keys only.
• Outdated CMS or plugin: update, remove what you do not use, enable automatic security updates (see First Steps).
• Exposed admin panel or database port: bind it to localhost, put it behind the firewall, or reach it over an SSH tunnel or tailnet.
• Leaked credential: rotate everywhere it was reused, and enable 2FA on MyNobull and the cloud portal.

5. Tell us what happened#

If there was an abuse report, reply in that ticket with what you found and fixed; that is what closes it. If there was not, a short note still helps: we may see the same pattern on other servers and can warn people. What we will not do is judge you for it. Everyone gets one; the difference is what you do next.