SSH (Secure SHell) is the ubiquitous tool for logging into and working on remote machines securely. All sensitive information is strongly encrypted, and in addition to the remote shell, SSH supports file transfer, and port forwarding for arbitrary protocols, allowing secure access to remote services. It replaces the classic telnet, rlogin, and similar non-secure tools - but SSH is not just a remote shell, it is a complete environment for working with remote systems.
In addition to the main ssh command, the SSH suite of programs includes tools such as scp (Secure Copy Program), sftp (Secure File Transfer Protocol), or ssh-agent to help with key management. The standard SSH port is port 22.
Several versions of SSH have existed. Today the most popular implementation of SSH, and de-facto standard, is OpenBSD's OpenSSH. This comes pre-installed on Gentoo, and is published under a BSD ("and freer") license.
SSH is multi-platform, and is very widely used: OpenSSH is installed by default on most Unix-like OSs, on Windows10, on MacOS, and can be installed on Android or "jailbroken" iOS (SSH clients are available). This makes SSH a great tool for working with heterogeneous systems.
Deployments of Gentoo Linux should already have OpenSSH installed, as the net-misc/openssh package is part of the system set. The presence and proper functioning of OpenSSH can be checked by running the ssh command, which should output a usage statement:
usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface] [-b bind_address] [-c cipher_spec] [-D [bind_address:]port] [-E log_file] [-e escape_char] [-F configfile] [-I pkcs11] [-i identity_file] [-J [user@]host[:port]] [-L address] [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port] [-Q query_option] [-R address] [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]] destination [command]
If no usage statement is printed, OpenSSH may be corrupt, or not installed. Try re-installation by following the emerge section, just as if rebuilding after a USE flag change. If OpenSSH were uninstalled, this should reinstall it. It should then remain installed, as part of the system set.
USE flags for net-misc/openssh Port of OpenBSD's free SSH release
||Add support for X11|
||Enable support for Linux audit subsystem using sys-process/audit|
||Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces|
||Add kerberos support|
||Use LDNS for DNSSEC/SSHFP validation.|
||Use the libedit library (replacement for readline)|
||Enable root password logins for live-cd environment.|
||Add support for PAM (Pluggable Authentication Modules) - DANGEROUS to arbitrarily flip|
||Build programs as Position Independent Executables (a security hardening technique)|
||Include builtin U2F/FIDO support|
||!!internal use only!! Security Enhanced Linux support, this must be set by the selinux profile or breakage will occur|
||Enable additional crypto algorithms via OpenSSL|
||!!do not set this during bootstrap!! Causes binaries to be statically linked instead of dynamically|
||Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)|
||Verify upstream signatures on distfiles|
||Enable XMSS post-quantum authentication algorithm|
emerge --ask --changed-use --oneshot net-misc/openssh
After changing any global USE flags in make.conf that affect the OpenSSH package, emerge world to update to the new USE flags:
emerge --ask --verbose --update --deep --newuse @world
In order to provide a secure shell, cryptographic keys are used to manage the encryption, decryption, and hashing functionalities offered by SSH.
On first run sshd init-script will generate all system keys if they does not exists. Usually there is no need to recreate them (unless the system has been compromised), but if such a need arises, this can be done manually through ssh-keygen command (rsa, ecdsa and Ed25519 algorithms):
ssh-keygen -t rsa -b 4096 -f /etc/ssh/ssh_host_rsa_key -N ""
ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ""
ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N ""
The SSH server is usually configured in the /etc/ssh/sshd_config file, though it is also possible to perform further configuration in OpenRC's /etc/conf.d/sshd, including changing the location of the configuration file. For detailed information on how to configure the server see the sshd_config man page.
The server provides means to validate its configuration using test mode:
Always validate the configuration changes prior restarting the service in order to keep the remote login available.
The ssh client and related programs (scp, sftp, etc.) can be configured using the following files:
SSH is a commonly attacked service. Tools such as sshguard and fail2ban monitor logs and black list remote users who have repeatedly attempted, yet failed to login. Utilize them as needed to secure a frequently attacked system.
Commands to run the SSH server will depend on active init system.
Add the OpenSSH daemon to the default runlevel:
rc-update add sshd default
Start the sshd daemon with:
rc-service sshd start
The OpenSSH server can be controlled like any other OpenRC-managed service:
rc-service sshd start
rc-service sshd stop
rc-service sshd restart
Active SSH connections to the server remain unaffected when issuing rc-service sshd restart.
To have the OpenSSH daemon start when the system starts:
systemctl enable sshd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/sshd.service to /usr/lib64/systemd/system/sshd.service.
To start the OpenSSH daemon now:
systemctl start sshd.service
To check if the service has started:
systemctl status sshd.service
OpenSSH provides several commands, see each command's man page for usage information:
- scp - secure file copy
- sftp - secure file transfer
- ssh-add - add private key identities to the authentication agent
- ssh-agent - authentication agent
- ssh-copy-id - use locally available keys to authorize logins on a remote machine
- ssh-keygen - authentication key utility
- ssh-keyscan - gather SSH public keys from servers
- sshd - OpenSSH daemon
During an active SSH session, pressing the tilde (~) key starts an escape sequence. Enter the following for a list of options:
Note that escapes are only recognized immediately after a newline. They may not always work with some shells, such as fish.
Passwordless authentication to a distant SSH server
On the client, if not already done, create a key pair. This can be done by running the following command (of course, not entering a passphrase):
ssh-keygen -t rsa
Generating public/private rsa key pair. Enter file in which to save the key (/home/larry/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/larry/.ssh/id_rsa. Your public key has been saved in /home/larry/.ssh/id_rsa.pub. The key fingerprint is: de:ad:be:ef:15:g0:0d:13:37:15:ad:cc:dd:ee:ff:61 larry@client The key's randomart image is: +--[ RSA 2048]----+ | | | . | | . .. n . | | . (: . . | | o . . : . | | . ..: >.> . | | * ?. . | | o.. .. .. | | :. . ! . | +-----------------+
Make sure an account for the user exists on the server, and then place the clients' id_rsa.pub file into the server's ~/.ssh/authorized_keys file in the user's home directory. This can be done by running the following command on the client computer (here, the user's passphrase on the server needs to be entered):
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/larry/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys larry@<server>'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '<server>'" and check to make sure that only the key(s) you wanted were added.
Afterwards a passwordless login should be possible doing:
Then on the server, the file /etc/ssh/sshd_config should be set to
Single machine testing
The above procedure can be tested out locally:
ssh-keygen -t rsa
Generating public/private rsa key pair. Enter file in which to save the key (/home/larry/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: ...
mv ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
Remote services over ssh
Copying files to a remote host
The OpenSSH 8.0 release notes, from 2019, state "The scp protocol is outdated, inflexible and not readily fixed. We recommend the use of more modern protocols like sftp and rsync for file transfer instead.". The OpenSSH 8.8 release notes, from 2021, state "A near-future release of OpenSSH will switch scp(1) from using the legacy scp/rcp protocol to using SFTP by default.".
OpenSSH comes with ssh-agent, a daemon to cache and prevent from frequent ssh password entries. When run, the environment variable SSH_AUTH_SOCK is used to point to ssh-agent's communication socket. The normal way to setup ssh-agent is to run it as the top most process of the user's session. Otherwise the environment variables will not be visible inside the session.
Depending on the way the graphical user session is configured to launch, it can be tricky to find a suitable way to launch ssh-agent. As an example for the lightdm display manager, edit and change /etc/lightdm/Xsession from:
exec ssh-agent $command
To tell ssh-agent the password once per session, either run
ssh-add manually or make use of the
Recent Xfce will start ssh-agent (and gpg-agent) automatically. If both are installed both will be started which makes identity management especially with SmartCards more complicated. Either stop XFCE from autostarting at least SSH's agent or disable both and use the shell, X-session or similar.
xfconf-query -c xfce4-session -p /startup/ssh-agent/enabled -n -t bool -s false
xfconf-query -c xfce4-session -p /startup/gpg-agent/enabled -n -t bool -s false
Terminal multiplexers to preserve sessions
It is possible to use a terminal multiplexer to resume a session after a dropped connection. Tmux and Screen are two popular multiplexers that can be used to be able to reconnect to a session, even if a command was running when the connection dropped out.
SSH over intermittent connections
When on unstable Internet connections, or when roaming between networks (such as when moving wifi networks), mosh can help avoid dropping SSH sessions.
Open new tabs for session with Kitty terminal
Kitty also provides other practical SSH functionality.
There are 3 different levels of debug modes that can help troubleshooting issues. With the
-v option SSH prints debugging messages about its progress. This is helpful in debugging connection, authentication, and configuration problems. Multiple
-v options increase the verbosity. Maximum verbosity is three levels deep.
ssh example.org -v
ssh example.org -vv
ssh example.org -vvv
Permissions are too open
An ssh connection will only work if the file permissions of the ~/.ssh directory and contents are correct.
- The ~/.ssh directory permissions should be 700 (drwx------), i.e. the owner has full access and no one else has any access.
- Under ~/.ssh:
- public key files' permissions should be 644 (-rw-r--r--), i.e. anyone may read the file, only the owner can write.
- all other files' permissions should be 600 (-rw-------), i.e. only the owner may read or write the file.
These permissions need to be correct on the client and server.
Death of long-lived connections
Many internet access devices perform Network Address Translation (NAT), a process that enables devices on a private network such as that typically found in a home or business place to access foreign networks, such as the internet, despite only having a single IP address on that network. Unfortunately, not all NAT devices are created equal, and some of them incorrectly close long-lived, occasional-use TCP connections such as those used by SSH. This is generally observable as a sudden inability to interact with the remote server, even though the ssh client program has not exited.
In order to resolve the issue, OpenSSH clients and servers can be configured to send a 'keep alive', or invisible message aimed at maintaining and confirming the live status of the link:
- To enable keep alive for all clients connecting to the local server, set
ClientAliveInterval 30(or some other value, in seconds) within the /etc/ssh/sshd_config file.
- To enable keep alive for all servers connected to by the local client, set
ServerAliveInterval 30(or some other value, in seconds) within the /etc/ssh/ssh_config or ~/.ssh/config file.
TCPKeepAlive noto help eliminate disconnections.
For example, to modify the server's configuration:
# The following ClientAlive values will keep an inactive session open for 30 minutes ClientAliveCountMax 60 ClientAliveInterval 30 # # Deactivate TCPKeepAlive TCPKeepAlive no
To modify the client's configuration:
# The following ServerAlive values will keep an inactive session open for 2 hours ServerAliveInterval 60 ServerAliveCountMax 120
New key does not get used
This scenario covers the case when a key to access a remote system has been created, the public key installed on the remote system, but the remote system is (for some reason) not accessible via ssh. This can happen if the name of the keyfile is not known to ssh.
Confirm which key files ssh is trying by running it with one of the verbose options, as described at the start of the Troubleshooting section. The verbose output will include the names of the keyfiles it is trying, and the one (if any) that actually gets used.
The default key files for the system are listed in the /etc/ssh/ssh_config, see the commented-out lines containing
There are several ways to use a key with a non-default name.
The key name can be specified on the command line every time:
ssh -i ~/.ssh/my_keyfile user@remotesys
Alternatively, modify the ssh configuration to add a special case for ssh to the remote system:
Host remotesys IdentityFile ~/.ssh/id_rsa IdentityFile ~/.ssh/my_keyfile
If any are specified, it appears to be necessary to specify all the desired keys on a remote host. Read up on the ssh IdentityFile.
X11 forwarding, not forwarding, or tunneling
Problem: After having made the necessary changes to the configuration files for permitting X11 forwarding, it is discovered X applications are executing on the server and are not being forwarded to the client.
Solution: What is likely occurring during SSH login into the remote server or host, the DISPLAY variable is either being unset or is being set after the SSH session sets it.
Test for this scenario perform the following after logging in remotely:
The output should be something similar to
localhost2.local:10.0 using server side
X11UseLocalhost no setting. If the usual
:0.0 is not displayed, check to make sure the DISPLAY variable within ~/.bash_profile is not being unset or re-initializing. If it is, remove or comment out any custom initialization of the DISPLAY variable to prevent the code in ~/.bash_profile from executing during a SSH login:
ssh -t larry@localhost2 bash --noprofile
Be sure to substitute
larry in the command above with the proper username.
A trick that works to complete this task would be to define an alias within the users' ~/.bashrc file.
- Autossh — a command that detects when SSH connections drop and automatically reconnects them.
- dropbear — a lightweight SSH server. It runs on a variety of POSIX-based platforms.
- Keychain — This document describes how to use SSH shared keys along with the keychain program.
- Mosh — a SSH client server that is aware of connectivity problems of the original SSH implementation.
- SCP — an interactive file transfer program, similar to the copy command, that copies files over an encrypted SSH transport.
- SFTP — an interactive file transfer program, similar to FTP, which performs all operations over an encrypted SSH transport.
- SSHFS — a secure shell client used to mount remote filesystems to local machines.
- SSH tunneling — a method of connecting to machines on the other side of a gateway machine.
- SSH jump host — employed as an alternative to SSH tunneling to access internal machines through a gateway.
- rsync — a powerful file sync program capable of efficient file transfers and directory synchronization.
- Securing the SSH service (Security Handbook)
- Starting the SSH daemon — Gentoo Handbook — Installation
- net-misc/connect — SSH Proxy Command -- connect.c
- https://lonesysadmin.net/2011/11/08/ssh-escape-sequences-aka-kill-dead-ssh-sessions/amp/ - A blog entry on escape sequences.
- https://hackaday.com/2017/10/18/practical-public-key-cryptography/ - Practical public key cryptography (Hackaday).
- https://www.akadia.com/services/ssh_putty.html - Port forwarding explained.