Protecting Linux servers from accidental connectivity loss

Introduction

Sometimes, you just login to your server to do regular administrative tasks, such as configuring applications, cron jobs, maybe move around some files manually, etc. These operations, even when being done over the Internet, are usually safe.

And sometimes, you login to your server to configure its network, e.g. update VPN configuration, firewall, modify routing or even interface configuration. In this case, you enter more dangerous commands such as:

ip route del default
ip rule add fwmark 0x420 lookup 69
ip route add default via 192.168.1.2 lookup 69
nft -f /etc/nftables.nft
nft flush ruleset
wg syncconf wg0
rc-service tailscale restart
ifdown eth0; ifup eth0

In that case, either everything goes according to your plan and you get a working system with a new configuration, or… You miscalculate something. You forgot some routing table on the system, some IP rule, misinterpreted your Nftables configuration or ran commands without anticipating what they actually did to your system. This all can happen to everyone of us. But what are the consequences? You may accidentally destroy your VPN, make it unavailable by firewall or severely screw up connection by a wrong routing table. Many of these accidents result in SSH session becoming unresponsive with further inability to login back again. At this point, options to recover the system are limited: go to the server and login with a keyboard, ask a friend or a datacenter employee to reboot your server, depending on its location. After reboot, you can use SSH again.

But what if… You didn’t have to reboot your server? What if it detects lost session and reboots itself?

Auto–reboot after a misconfiguration would save lots of nerves and hassle of everyone involved. And as it turns out, Linux is flexible enough to allow to do this with a couple of scripts. Wanna know how? Then this article’s for you.

All technical instructions in this article are tested for Alpine Linux 3.24. However, the ideas and principles stated in this article can be used to implement similar solutions on other Linux distribution as well.

Verifying that the server is still accessible via SSH

First, a server needs a way to know whether or not it can still be accessed over SSH. One very robust way to verify that is to try connecting to the server from different hosts. This means that the solution requires a number of hosts from which admins are expected to use SSH.

The procedure is, for the most part, standard SSH configuration:

(On a client) Create a user watchdog.
```
adduser -S -h /home/watchdog watchdog
```

Generate an SSH key for this user, without passphrase.

mkdir /home/watchdog/.ssh
ssh-keygen -f /home/watchdog/.ssh/id_ed25519
chown -R watchdog /home/watchdog/.ssh

Copy SSH host public key from server to client’s known hosts.
```
ssh server 'cat /etc/ssh/ssh_host_ed25519_key.pub' >> /home/watchdog/.ssh/known_hosts
```
Replace server with real IP address/hostname in your network. This applies to every further example.
Schedule a connection to server periodically (for example, every 2 minutes).
```
crontab -e -u watchdog
```
Add this entry:
```
*/2 * * * * ssh server
```

And there is one trick that makes this procedure secure. Do not let client specify command to run on a server. All the client needs to do is to just connect; the server will decide itself what to do on this connection.

The server configuration is a little more interesting.

Create a user watchdog.
```
adduser -h /home/watchdog watchdog
```
Add the following configuration to /etc/ssh/sshd_config:
```
1Match User watchdog
2	ForceCommand touch /var/tmp/do-not-reboot-me
3 DisableForwarding yes
4 PermitTTY no
5	LogLevel quiet
```
Line 2 will force the connected client to only run touch /var/tmp/do-not-reboot-me, regardless of what command client specified^[2]. Lines 3 and 4 disable other SSH client features set on by default — port forwarding and TTY. Both of them watchdog clients don’t need. Line 5 just exists to avoid flooding your logs with logins every 2 minutes.
Reload SSH daemon.
```
rc-service sshd reload
```
Copy client’s public key to server’s authorized keys.
```
ssh client 'cat /home/watchdog/.ssh/id_ed25519.pub' >> /home/watchdog/.ssh/authorized_keys
```
Replace client with real IP address/hostname in your network. This applies to every further example.

And it’s done. Clients will regularly connect to server, which will prompt it to create do-not-reboot-me file. Leave both server and client alone for 2 minutes and you’ll get that file on a server. That’s the way clients will signal the server that it’s still reachable via SSH.

Rebooting the system after it became unreachable

Clients can now signal a server that SSH is accessible at this moment of time. The next thing to do is to make use of do-not-reboot-me file. And for that, the following script:

1#!/usr/bin/env sh
2rm /var/tmp/do-not-reboot-me && exit
3touch /var/tmp/do-not-reboot-me
4touch /var/tmp/watchdog-rebooted-me
5echo "Rebooted due to SSH not being available at $(date)." >> /etc/motd
6reboot

Will effectively consume do-not-reboot-me file, but if it doesn’t exist already, reboot the system.

Line 3 may seem unusual. At line 2, I removed do-not-reboot-me, but then restoring it again before going reboot. This is a protection against one particular problem that arises when executing this kind of scripts in Cron.

To schedule tasks, Cron counts time from “wall clock”, not uptime. In other words, you don’t schedule tasks to execute “every 15 minutes since boot”. You schedule “execute every 15th minute”. For example, if a server boots at 19:43, cron job will trigger first time after 2 minutes, then second time after 15. This can get the system into a situation where, even after a clean reboot, a new reboot triggers not because clients couldn’t access the server — but because server got unrealistically impatient for them.

By putting a do-not-reboot-me file, I can ensure that next reboot will never have first cron activation reboot the system again. That way, clients have 15-30 minutes to connect, not 0-15. According to FHS, /var/tmp’s contents persist after reboot, which is just what this script needs^[1].
Line 4 leaves another empty file in /var/tmp. This file will be important later.
Line 5 also writes notification to MotD (Message of the Day), which will immediately remind next admin logging in what happened with the system.

If that script executes periodically in Cron (let’s say, every 15 minutes):

*/15 * * * * /etc/periodic/15min/watchdog

Then the system will reboot if for the last 15 minutes no watchdog client has authenticated over SSH. While clients are online and connect via SSH periodically, server never reboots. But if SSH is disrupted for 15 minutes for any reasons, clients can’t reach the server, server won’t touch do-not-reboot-me file and will eventually reboot.

Always set interval for watchdog server longer than for watchdog clients.

Booting different set of services after reboot

So far, this recovery system successfully recovers itself after temporary changes, such as accidental commands without configuration written to /. This system still won’t save from permanent breaking changes — for example, writing Nftables which suddenly makes SSH inaccessible. And this is a much harder problem, because a simple reboot isn’t enough. What to do in this case?

For a solution, I propose on the next reboot disabling every service other than those needed to establish SSH connection. That way, if a firewall configuration disrupts SSH, it simply won’t get loaded on the next reboot. However, it is still possible to misconfigure the SSH daemon itself, so… Be careful.

I could try to think of a system which rolls back SSH as well, but won’t, and here’s why. It is important to understand that as long as you manage the rollback system, you can always screw the rollback system as well. That’s why my solution doesn’t make the system 100% bulletproof from whatever I do on the system. It significantly reduces the possibility of irrecoverable errors. This is also why I have been aiming for simplicity of code and as few dependencies as possible — sh and cron should be all it takes.

OpenRC has runlevels. So why not create a simple runlevel which will replace default when system initializes after the watchdog brought it down?

Create a recovery runlevel and add only sshd to it.

mkdir /etc/runlevels/recovery
rc-update add sshd recovery

You can also add additional services which your server needs to be reachable via SSH. For example, frp if the server’s behind NAT.

Make sure crond starts only at default runlevel:

rc-update del crond sysinit
rc-update del crond boot
rc-update del crond recovery
rc-update add crond default

The next step is to configure the init system’s boot order, i.e. the order of execution of runlevels. By default, Alpine Linux uses Busybox init. Busybox init reads /etc/inittab for configuration. So I decided to edit it.

Find the following 3 lines, in order, in /etc/inittab:

::sysinit:/sbin/openrc sysinit
::sysinit:/sbin/openrc boot
::wait:/sbin/openrc default

This is the order in which init launches OpenRC, runlevel-by-runlevel. Levels sysinit and boot should be left intact, but runlevel default should be conditionally substituted by recovery.

Write the script that consumes the file generated by a cron job and chooses between booting either recovery or default.

#!/usr/bin/env sh
if rm /var/tmp/watchdog-rebooted-me 2>/dev/null
then
	/sbin/openrc recovery
else
	/sbin/openrc default
fi

Put that file in, for example, /recovery/openrc-handler.

Then change the third line in /etc/inittab to get these 3 lines instead:

::sysinit:/sbin/openrc sysinit
::sysinit:/sbin/openrc boot
::wait:/recovery/openrc-handler

Done. Now, even if you do:

rc-update del sshd default
rc-service sshd stop

Then, after 15 minutes, system reboots into recovery runlevel with SSH enabled, allowing you to recover the system back:

rc-update add sshd default
reboot

Just be extra careful around recovery runlevel:

rc-update del sshd recovery  # Disables the ability to recover after config write

Conclusion

The recovery system I’ve designed is essentially a software watchdog, but built with not a separate daemon, but plain scripting. There are very few lines of code, which makes the system easily understandable. The code is written in POSIX–compatible shell, which makes this system portable across Linux distributions. The system is secured, allowing clients to use SSH, but not to execute arbitrary commands. Finally, this system covers 2 failure scenarios: accidental temporary change and accidental permanent change.

Solutions like this are what allows remote server configuration to be a lot less stressful operation.

References

LSB Workgroup. The Linux Foundation. "Filesystem Hierarchy Standard" (2015-06-03), version 3.0.
GNU. "sshd_config(5)" (2026-03-28).