How to secure a root server – Marvin Haagens Tech Blog

In this article I will show you which steps you need to do to secure your root server. It consists of the general practises of setting up and the detailed implementation on Debian Buster.

Change the SSH port to a custom one

Most attackers try to find an open SSH port on the default port number (which is 22) and will pass by if there is no open port. So you definitely should change the port number of your SSH connection.

Open /etc/ssh/sshd_config with your favorite editor and set following line:

Port XXXX

XXXX is the placeholder for the portnumber you want the ssh server to listen for connections on.

Now restart the SSH daemon service:

service ssh restart

Don’t close the existing SSH connection and try to login with a new SSH connection using the new configuration. If you can successfully login you can close the previous SSH connection.

Setup cooldown after failed logins

To prevent Bruteforce and Dictionary attacks you should setup the server to reject logins for a user account for some time if the user failed a given amount of login attempts in a row.

You can achieve this by using fail2ban. Run

apt-get install fail2ban

to install it. Check that it is running:

systemctl status fail2ban.service

the output should look like this:

● fail2ban.service - Fail2Ban Service
   Loaded: loaded (/lib/systemd/system/fail2ban.service; enabled; vendor preset:
   Active: active (running) since Sun 2022-08-28 23:10:03 CEST; 7min ago
     Docs: man:fail2ban(1)
 Main PID: 971 (fail2ban-server)
    Tasks: 3 (limit: 105)
   Memory: 14.0M
   CGroup: /system.slice/fail2ban.service
           └─971 /usr/bin/python3 /usr/bin/fail2ban-server -xf start

Configure Fail2Ban

To configure fail2ban create a file in /etc/fail2ban/jail.d by copying /etc/fail2ban/jail.conf:

cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.d/temperary-block-account-on-failed-logins.local

You should not edit jail.conf as it is managed by fail2ban itself and might be overwritten by later package updates

Now open the created file with your favorite text editor and edit the rules according to your needs:

nano /etc/fail2ban/jail.d/temperary-block-account-on-failed-logins.local

Remove the warning that says that you should not modify that file to avoid confusions when you need to edit the file on a later point in time, as it relates to the jail.conf file, not the files within jail.d.

Also don’t forget to active Fail2Ban for the sshd by uncommenting the line

enabled = true

within the sshd section of your created rule file.

The default settings block an ip address on all ports for 10 minutes after 5 failed login attempts within the last 10 minutes.

For more details on the settings within the rule file you can take a look on this links:

After finishing the rule configuration you need to restart Fail2Ban:

systemctl restart fail2ban

Send email notification on failed login attemps

If an IP address gets blocked for too many failed login attempts the server should notify you by sending an email to you for further investigation.

First install Sendmail:

apt-get install sendmail sendmail-cf m4

And configure it:

sendmailconfig

The default settings that are shown should be fine in most cases.

The default config is already fine for sending emails. If you don’t want sendmail to handle incoming emails you should close all unneeded ports except for port 25. Take a look at the section „Close unneeded ports“ on how to achieve this.

You can find the sendmail configuration file at /etc/mail/sendmail.mc

Now to setup fail2ban to send the notification email when an account is banned, open the configuration file you created in the previous step:

nano /etc/fail2ban/jail.d/temperary-block-account-on-failed-logins.local

Look for the line defining the „action“ key:

action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

And change it to:

action = %(action_mwl)s

action = %(action_mw)s

Depending on how much detail about the incident you want the notification email to contain.

Now look for the line defining the „destemail“ key:

destemail = root@root

And change its value to the email address you want the notification email to be sent to.

Also look for the „sender“ key:

sender = root@<fq-hostname>

And change its hostname (the part behind the @) to the full qualified domain name of the machine which will send the notification email.

Now restart fail2ban to activate your new config settings and start the fail2ban client:

systemctl restart fail2ban
fail2ban-client start

You now should have received a notification email on the configurated email address stating that the server has started correctly.

Now you can check that everything works correctly by opening another SSH terminal and try to connect to the server with wrong credentials 6 times.

Don’t close your current SSH terminal. If Fail2Ban is set up correctly you will need that connection to unban your IP address if you don’t want to wait for the ban timeout (10 minutes per default).

You now should have received an email notifying you that your IP address has been blocked. You can list all currently blocked IPs by using your existing SSH connection:

fail2ban-client status sshd

The output should look similar to this:

Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     5
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 1
   |- Total banned:     1
   `- Banned IP list:   xxx.xxx.xxx.xxx

Now run the following command to unblock the IP manually:

fail2ban-client set <jailname> unbanip <ipaddress>

In the example jailname is „sshd“ and IP address is the IP address you want to unblock.

Close unneeded ports

You should close all ports in your firewall except those which are used by services you want to be available to the public. This significantly reduces the amount of services that might be attacked.

For this, you need to install UFW:

apt-get install ufw

Block all incoming traffic:

ufw default deny incoming

Allow all outgoing traffic:

ufw default allow outgoing

Now allow all ports where your server shall listen to incoming traffic. At least you need to allow the SSH port to be able to access the server:

ufw allow xxxx

You need to run this command for every service you wish to allow to be accessed from external systems.

xxxx is the placeholder for the port number of the service you want to allow to be accessed

Now enable UFW:

ufw enable

To check if all ports are closed correctly you can run a port scanner such as Nmap. For Windows I suggest using Zenmap, a graphical frontend for Nmap:

The result of a port scan with Zenmap (Nmap GUI)

For more details you can take a look at UFW tutorial at digitalocean.com.

Send notification email on SSH login

You should configure your server to send a notification email when someone successfully logs in via SSH so that you will know if your server has been hacked. For this, first create a script „notify-ssh-login.sh“:

if [ -n "$SSH_CLIENT" ]; then
{
  IPADDR="$(echo $SSH_CLIENT | awk '{print $1}')"
  echo "Subject: [$$CATEGORY$$] SSH Login: ${USER} from ${IPADDR}"
  echo ""
  echo "User ${USER} logged in via SSH from ${IPADDR} onto $(hostname -f)".
} | sendmail -f "$$CATEGORY$$ <root@$$HOSTNAME$$>" $$RECIPIENT$$
fi

Replace $$HOSTNAME$$ with the hostname of your Gitlab server, $$CATEGORY$$ with the category (for example „Gitlab Runner“) and $$RECIPIENT$$ with the mail address to send the log to.

Now set the x-bit to make it executable:

chmod +x notify-ssh-login.sh

Now run the script to test it. You should receive an email that not yet contain location data, since it was not called while an SSH login.

Now set a symlink in /etc/profile.d to your notify-ssh-login.sh to run the script when a user logs in:

cd /etc/profile.d
ln -s ~/notify-ssh-login.sh

You will now be notified via email when someone logs in to your server

Keep your server up to date

You need to regularly update all services and applications that you run on the server to ensure, that possible critical vulnerabilities are getting patched as soon as possible.

First update the package list:

apt-get update

And then install all updates:

apt-get upgrade

You need will be asked if you want to install the packages. Input „y“ followed by the return key to ackknowledge.

Maybe you want to automatically update the server. For this, first create the script file update-server.sh at a location you like with following content:

#!/bin/bash
LOGFILE=update-server.log
cd "$(dirname "$0")"
echo "Logs:" > $LOGFILE
echo "Working directory set to $(dirname "$0")" | tee -a $LOGFILE
(apt-get update && apt-get upgrade -y) 2>&1 | sed 's/^/  /' | tee -a $LOGFILE
{
  echo "Subject: [$$CATEGORY$$] Server update report"
  echo ""
  echo "$(cat $LOGFILE)"
} | sendmail -f "$$CATEGORY$$ <root@$$HOSTNAME$$>" $$RECIPIENT$$

Replace $$HOSTNAME$$ with the hostname of your Gitlab server, $$CATEGORY$$ with the category (for example „Gitlab Runner“) and $$RECIPIENT$$ with the mail address to send the log to.

Now set the x-bit to make it executable:

chmod +x update-server.sh

Now test run the script. You should receive an email with the logs.

Now open the /etc/crontab file and add following line:

  0  0  *  *  * root      /srv/apt/update-server.sh

Your system is now set up to automatically pull updates

When you run an apt-get command and you get a message indicating that the lock for /var/lib/dpkg/lock could not be acquired dpkg probably showed a configuration dialog in the previous run of the update script. To make that dialog visible execute „dpkg –configure -a“

The update could get stuck when an upgrade provides a new version of a config file that you have changed manually, because Dpkg in this case will ask what to do. To prevent this error from occuring, you can create a file „70changedconffile“ in /etc/apt/apt.conf.d:

Dpkg::Options {
  "--force-confdef";
  "--force-confold";
}

This will ensure that Dpkg will automatically keep the manually changed config files without further asking.