troubleshooting a ssh.socket problem

From the “wasting hours with computer problems” department:

Today I updated a few packages on my Ubuntu based home server as I had done countless times before. One of the packages was openssh-server, again a package that I had updated countless times before without any problems. But today is a special day. The server rebooted and everything seemed to work fine, until I tried to ssh into it. I got the dreaded “Connection refused” error in Putty.

Now try to search the web for that. You will find a lot of pages mentioning that error together with ssh and Ubuntu and most of them will tell you to install the ssh server package, configure it correctly and make sure the service has been started.

Some actually take into account that Ubuntu switched to starting ssh through ssh.socket rather than ssh.service and tell you to revert that switch if you get the error. While that does work, it does not say why the default way of using ssh.socket does not work, so I tried to dig deeper.

There are some sites that tell you how to change the default ssh port from 22 to something else. They mention that you no longer do that by changing /etc/ssh/sshd_config but by creating a file in /etc/systemd/system/ssh.socket.d . Others mention that in later Ubuntu versions the socket configuration in sshd_config is now used again.

I looked into /etc/systemd/system/ssh.socket.d and found a file override.conf there with the content:

[Socket]
# first delete the previous value
ListenStream=
# then set the new value (22)
ListenStream=22

Apparently somebody (maybe long ago me, or maybe the package maintainer) had tried to force the socket to 22 for whatever reason.

I moved that file away, made sure to enable ssh.socket and disable ssh.service, reloaded the daemons …

sudo systemctl enable --now ssh.socket
sudo systemctl disable --now ssh.service
sudo systemctl daemon-reload

… and tried to connect via ssh again. It worked! Just to be sure, I rebooted the computer and connecting via ssh still worked.

Unfortunately I still don’t have a clue how to troubleshoot the problem. This was just some lucky guess. But at least it works now.

EDIT:
See the corresponding bug report for Ubuntu for some more info.

Did I ever mention that I hate computers? I should have learned something useful and become a gardener like my late father.

Oh, btw. AI was useless as always in this case. As you would expect (if you know how it works), it regurgitated all the wrong and misleading hints I had already found on the web.