Advertisement

I encountered persistent issues with my container image build pipeline. The pipeline is run by my local GitLab CI instance. After struggling for a considerable period, I decided to address the problem by transitioning from a build setup relying on “docker” to a new setup built on “podman/buildah”. This required a complete overhaul of the build process. Unfortunately, this change broke the deployment of my website using GitLab CI.

During my investigation of the issue, I realized the need to update the information in the ~/.ssh/known_hosts file used during the GitLab CI deployment. To accomplish this, I did a research to identify the most suitable tools for the job. This article is about the tools found.

Requisites for readers

This article is intended for individuals with a fundamental understanding of the Linux operating system. It is written with the assumption that the reader does not have Arch Linux installed on their workstation. The instructions provided in this article should be applicable to most Linux distributions.

I added tags to clarify which commands have to be executed on which of your systems:

  • Workstation: Your local desktop computer or laptop which you use for daily business.
  • Server: Your SSH server.

To enhance readability, I have omitted the use of the sudo prefix for commands executed as root. Instead, I’ve utilized the following syntax for the commands in this article. However, for your regular usage, I strongly recommend incorporating the sudo command for proper execution and authorization.

  • $ command : Running the command as a normal or admin user
  • # command : Running the command as root

Generate the entries

To create the necessary directory and set permissions and ownership, please run the following command:

Workstation Create ~/.ssh directory

$ mkdir ~/.ssh
$ chown "$(id -r -u):$(id -r -u)" ~/.ssh
$ chmod 0700 ~/.ssh

Workstation Verify permissions and ownership of the ~/.ssh directory

$ ls -al -d ~/.ssh

drwx------ 2 <user> <user> 4096 Nov 14  2022 /home/<user>/.ssh

Workstation Generate entry to be added to the ~/.ssh/known_hosts file.

To get an idea about the structure of an entry in the file, please run the following command. This outputs all entries for the server.example.com in the hashed format (-H).

$ ssh-keyscan -H server.example.com

# server.example.com:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
|1|il/kh5SdrgkYNWV8TSwVMz1TRcw=|NM4p6a8UXDEtpC8V4bIKIX/bwEE= ecdsa-sha2-nistp256 BBBBE2VjZFNhLXNoYTItbmlzdFByNTYBBBBIbmlzdFByNTYBBBBBBB4elFx0CRRVrgUxbe8FNsIqiJjb493j/xYrhg5GqMTiYQ2L+hyNX1Jqithul+zwPqrEWOWGL4kGjpsUSBZ49uw=
# server.example.com:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
|1|B1C+bJy7zcSM1ucu+dS9iBo/SxU=|LFjiNSishb1OXF2YjFJKnMNFPVo= ssh-rsa BBBBB3NzaC1yc2EBBBBBIwBBBQEByZw74pETymcIUpmQ5c0Vdd/DgLcEt518TDyFZJ7dLICmJomk5Mqm6X8Dhjundr79ozIqBuXmLBlh5L1ff0ibbZXiVBokU63Soxk4diXEojaXDMcc3CXZL89wM2hKuJIlVGiNvG8M9QGavq5tgsfyP9W4604IEMLns5EdYX/eFFfGlvnz/phtCBlTJiQRklmgwyBszJg2CF83Zsscv4FOy4Zjkq2aZuSgFhBKy5JacGY/XeRB2dhvi03TXMl4sGPBV/+Z9hydyMb/qUhSxa1FznSzr3+ZNNB01jPZ6V/bOr3mXY+dj3+27Ut43gmoox/xj4IBhflXGCOgBZk7KfVvnQ==
# server.example.com:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
|1|BzixOn5ZmGmhB4qIgirIoFC646Q=|k83iNqtTtsbbxD8bDl5f4WF4I6c= ssh-ed25519 BBBBC3NzaC1lZDI1NTE5BBBBIIo2pPr9GFKwgFBRF3UuqeaxYP8yV2T2Ps2F8Y9kUG6n
# server.example.com:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
# server.example.com:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5

If your SSH server listens on a non-standard port (e.g. tcp port 2222), use -p <port>.

$ ssh-keyscan -p 2222 -H server.example.com

Workstation Add entry to the ~/.ssh/known_hosts file.

Important: Make sure to use >> for output redirection, otherwise you might overwrite your existing ~/.ssh/known_hosts file.

$ ssh-keyscan -H server.example.com >> ~/.ssh/known_hosts

Find entry for host

Workstation Find entry with the standard SSH port used

$ ssh-keygen -H -F server.example.com -f ~/.ssh/known_hosts

# Host server.example.com found: line 8
|1|il/kh5SdrgkYNWV8TSwVMz1TRcw=|NM4p6a8UXDEtpC8V4bIKIX/bwEE= ecdsa-sha2-nistp256 BBBBE2VjZFNhLXNoYTItbmlzdFByNTYBBBBIbmlzdFByNTYBBBBBBB4elFx0CRRVrgUxbe8FNsIqiJjb493j/xYrhg5GqMTiYQ2L+hyNX1Jqithul+zwPqrEWOWGL4kGjpsUSBZ49uw=
# Host server.example.com found: line 9
|1|B1C+bJy7zcSM1ucu+dS9iBo/SxU=|LFjiNSishb1OXF2YjFJKnMNFPVo= ssh-rsa BBBBB3NzaC1yc2EBBBBBIwBBBQEByZw74pETymcIUpmQ5c0Vdd/DgLcEt518TDyFZJ7dLICmJomk5Mqm6X8Dhjundr79ozIqBuXmLBlh5L1ff0ibbZXiVBokU63Soxk4diXEojaXDMcc3CXZL89wM2hKuJIlVGiNvG8M9QGavq5tgsfyP9W4604IEMLns5EdYX/eFFfGlvnz/phtCBlTJiQRklmgwyBszJg2CF83Zsscv4FOy4Zjkq2aZuSgFhBKy5JacGY/XeRB2dhvi03TXMl4sGPBV/+Z9hydyMb/qUhSxa1FznSzr3+ZNNB01jPZ6V/bOr3mXY+dj3+27Ut43gmoox/xj4IBhflXGCOgBZk7KfVvnQ==
# Host server.example.com found: line 10
|1|BzixOn5ZmGmhB4qIgirIoFC646Q=|k83iNqtTtsbbxD8bDl5f4WF4I6c= ssh-ed25519 BBBBC3NzaC1lZDI1NTE5BBBBIIo2pPr9GFKwgFBRF3UuqeaxYP8yV2T2Ps2F8Y9kUG6n

Workstation Find entry with a non-standard SSH port used

$ ssh-keygen -H -F "[server.example.com]:2222" -f ~/.ssh/known_hosts

# Host [server.example.com]:2222 found: line 8
|1|il/kh5SdrgkYNWV8TSwVMz1TRcw=|NM4p6a8UXDEtpC8V4bIKIX/bwEE= ecdsa-sha2-nistp256 BBBBE2VjZFNhLXNoYTItbmlzdFByNTYBBBBIbmlzdFByNTYBBBBBBB4elFx0CRRVrgUxbe8FNsIqiJjb493j/xYrhg5GqMTiYQ2L+hyNX1Jqithul+zwPqrEWOWGL4kGjpsUSBZ49uw=
# Host [server.example.com]:2222 found: line 9
|1|B1C+bJy7zcSM1ucu+dS9iBo/SxU=|LFjiNSishb1OXF2YjFJKnMNFPVo= ssh-rsa BBBBB3NzaC1yc2EBBBBBIwBBBQEByZw74pETymcIUpmQ5c0Vdd/DgLcEt518TDyFZJ7dLICmJomk5Mqm6X8Dhjundr79ozIqBuXmLBlh5L1ff0ibbZXiVBokU63Soxk4diXEojaXDMcc3CXZL89wM2hKuJIlVGiNvG8M9QGavq5tgsfyP9W4604IEMLns5EdYX/eFFfGlvnz/phtCBlTJiQRklmgwyBszJg2CF83Zsscv4FOy4Zjkq2aZuSgFhBKy5JacGY/XeRB2dhvi03TXMl4sGPBV/+Z9hydyMb/qUhSxa1FznSzr3+ZNNB01jPZ6V/bOr3mXY+dj3+27Ut43gmoox/xj4IBhflXGCOgBZk7KfVvnQ==
# Host [server.example.com]:2222 found: line 10
|1|BzixOn5ZmGmhB4qIgirIoFC646Q=|k83iNqtTtsbbxD8bDl5f4WF4I6c= ssh-ed25519 BBBBC3NzaC1lZDI1NTE5BBBBIIo2pPr9GFKwgFBRF3UuqeaxYP8yV2T2Ps2F8Y9kUG6n

Remove entries

Workstation Remove entry for a single host

It is crucial to exercise caution if an error occurs during the SSH handshake with one of your production servers, especially when hostkey changes are not expected (which is typically the case).

It’s important to investigate and address any errors during the SSH handshake promptly, as they may indicate potential security risks. Monitoring and verifying the hostkey information is essential in maintaining the integrity and security of your production servers.

Occasionally, when using virtual systems for testing purposes, there might be instances where hostnames are reused. This can lead to warnings or connection aborts when attempting to connect to an SSH server with a different hostkey, depending on your SSH setup. To resolve this issue for your test machines, you can remove the hostkey entry from the ~/.ssh/known_hosts file.

$ ssh-keygen -R server.example.com -f ~/.ssh/known_hosts

Conclusion

It’s simple as that. I hope this article helps your to improve the maintenance of your ~/.ssh/known_hosts file.

References

Discussion

If you found a mistake in this article or would like to contribute some content to it, please file an issue in this Git Repository

Disclaimer

The contents of this article are put together to the best of the authors' knowledge, but it cannot be guaranteed that it's always accurate in any environment. It is up to the reader to make sure that all information found in this article, does not do any damage to the reader's working environment or wherever this information is applied to. In no event shall the authors or copyright holders be liable for any claim, damages or other liability, arising from, out of or in connection with this article. Please also note the information given on the Licences' page.