Skip to main content
DevOpsKit.in
Linux·9 min read

SSH Key Setup — Generate, Configure & Recover When You're Locked Out

Set up SSH key authentication for passwordless server access. Covers key generation, ssh-agent, config files, multi-key management, and fixing common permission errors.

DT

DevOps Engineer & Technical Writer

The Problem

SSH Key Authentication Flow Client Private Key ~/.ssh/id_ed25519 SSH Handshake 1. Client sends public key ID 2. Server sends challenge 3. Client signs with private key 4. Server verifies signature Server authorized_keys Public Key stored Session Established Encrypted channel — no password sent Private = Never share Public = Share freely

You are typing passwords every time you SSH into a server. Or worse, your team is sharing passwords in Slack. SSH key authentication eliminates passwords, provides stronger security, enables automation, and lets you manage access per-key instead of per-password.

But SSH is notoriously picky about file permissions, key formats, and configuration. One wrong permission bit and authentication silently falls back to password mode.

Generating SSH Keys

Ed25519 (recommended)

ssh-keygen -t ed25519 -C "yourname@company.com"

RSA (for compatibility with older systems)

ssh-keygen -t rsa -b 4096 -C "yourname@company.com"

Non-interactive generation (for automation)

ssh-keygen -t ed25519 -f ~/.ssh/deploy_key -N "" -C "deploy@ci-server"

Generated files

~/.ssh/id_ed25519       # Private key (NEVER share this)

~/.ssh/id_ed25519.pub   # Public key (this goes on servers)

Copying Your Public Key to a Server

Method 1: ssh-copy-id

ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server.example.com

Method 2: Manual copy

cat ~/.ssh/id_ed25519.pub | ssh user@server.example.com "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Method 3: For cloud instances

aws ec2-instance-connect send-ssh-public-key \

  --instance-id i-0abc123 \


  --instance-os-user ec2-user \


  --ssh-public-key file://~/.ssh/id_ed25519.pub

SSH Config File

The ~/.ssh/config file eliminates repetitive command-line options:

# ~/.ssh/config


Host prod-web-*


    User deploy


    IdentityFile ~/.ssh/deploy_ed25519


    StrictHostKeyChecking yes


    Port 22



Host prod-web-1


    HostName 10.0.1.10



Host prod-web-2


    HostName 10.0.1.11



Host bastion


    HostName bastion.example.com


    User admin


    IdentityFile ~/.ssh/bastion_key


    ForwardAgent yes



Host internal-*


    ProxyJump bastion


    User deploy


    IdentityFile ~/.ssh/internal_key



Host internal-db


    HostName 10.0.2.50



Host internal-api


    HostName 10.0.2.51



Host github.com


    IdentityFile ~/.ssh/github_ed25519


    IdentitiesOnly yes



Host *


    AddKeysToAgent yes


    ServerAliveInterval 60


    ServerAliveCountMax 3


    TCPKeepAlive yes

Now connect with just:

ssh prod-web-1

ssh internal-db     # Automatically jumps through bastion

SSH Agent

The agent holds decrypted private keys in memory:

eval "$(ssh-agent -s)"

ssh-add ~/.ssh/id_ed25519


ssh-add -l              # List loaded keys


ssh-add -D              # Remove all keys


ssh-add -t 4h ~/.ssh/id_ed25519   # Auto-remove after 4 hours

macOS keychain integration

ssh-add --apple-use-keychain ~/.ssh/id_ed25519

Add to ~/.ssh/config:

Host *

    UseKeychain yes


    AddKeysToAgent yes

Managing Multiple Keys

ssh-keygen -t ed25519 -f ~/.ssh/work_ed25519 -C "name@work.com"

ssh-keygen -t ed25519 -f ~/.ssh/personal_ed25519 -C "name@personal.com"


ssh-keygen -t ed25519 -f ~/.ssh/deploy_ed25519 -C "deploy@ci" -N ""

Configure in ~/.ssh/config:

Host github.com-work

    HostName github.com


    User git


    IdentityFile ~/.ssh/work_ed25519


    IdentitiesOnly yes



Host github.com-personal


    HostName github.com


    User git


    IdentityFile ~/.ssh/personal_ed25519


    IdentitiesOnly yes

Then in git repositories:

git remote set-url origin git@github.com-work:company/repo.git

git remote set-url origin git@github.com-personal:username/repo.git

Server-Side Configuration

Edit /etc/ssh/sshd_config:

PubkeyAuthentication yes

PasswordAuthentication no


PermitRootLogin prohibit-password


AuthorizedKeysFile .ssh/authorized_keys


MaxAuthTries 3


ClientAliveInterval 300


ClientAliveCountMax 2


AllowUsers deploy admin


AllowGroups ssh-users

After editing:

sudo sshd -t                    # Validate config

sudo systemctl reload sshd      # Apply changes

Required Permissions

SSH is strict about file permissions:

# Local machine

chmod 700 ~/.ssh


chmod 600 ~/.ssh/id_ed25519


chmod 644 ~/.ssh/id_ed25519.pub


chmod 600 ~/.ssh/config



# Remote server


chmod 700 ~/.ssh


chmod 600 ~/.ssh/authorized_keys


chmod 755 ~

Fix permissions script

chmod 700 ~/.ssh

find ~/.ssh -type f -name "*.pub" -exec chmod 644 {} \;


find ~/.ssh -type f ! -name "*.pub" -exec chmod 600 {} \;

Troubleshooting

Debug connection

ssh -v user@server       # Basic verbosity

ssh -vvv user@server     # Maximum verbosity

Common issues and fixes

# "Permission denied (publickey)"

# Check 1: Is public key in authorized_keys?


ssh user@server "cat ~/.ssh/authorized_keys"



# Check 2: Are permissions correct?


ssh user@server "ls -la ~/.ssh"



# Check 3: Is the correct key being offered?


ssh -v user@server 2>&1 | grep "Offering"



# "Too many authentication failures"


ssh -i ~/.ssh/specific_key -o IdentitiesOnly=yes user@server



# Key works manually but not in script/cron


ssh -i /home/user/.ssh/deploy_key -o BatchMode=yes user@server "command"

Check server-side auth logs

sudo journalctl -u sshd --since "5 min ago"

sudo tail -f /var/log/auth.log        # Debian/Ubuntu


sudo tail -f /var/log/secure           # RHEL/CentOS

Key Rotation

# Generate new key

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_new -C "rotated $(date +%Y-%m)"



# Add new key to servers


ssh-copy-id -i ~/.ssh/id_ed25519_new.pub user@server



# Test new key


ssh -i ~/.ssh/id_ed25519_new user@server



# Remove old key from servers


ssh user@server "sed -i '/old_key_comment/d' ~/.ssh/authorized_keys"



# Replace local key


mv ~/.ssh/id_ed25519_new ~/.ssh/id_ed25519


mv ~/.ssh/id_ed25519_new.pub ~/.ssh/id_ed25519.pub

Common Mistakes

  • Wrong permissions on ~/.ssh or key files — SSH silently ignores keys if permissions are too open. Private keys must be 600, directory must be 700.
  • Disabling password auth before testing key auth — Always verify key login works in a separate session first.
  • Using IdentitiesOnly yes without IdentityFile — This tells SSH to only try specified keys, but if none is specified, it tries nothing.
  • Forgetting ForwardAgent for jump hosts — Without agent forwarding, you cannot authenticate to internal servers through a bastion.
  • Leaving deploy keys without passphrases on laptops — Deploy keys without passphrases are for servers only. Personal keys should always have a passphrase.
  • Not checking SELinux contexts on RHEL — SELinux can prevent sshd from reading authorized_keys. Run restorecon -Rv ~/.ssh.

    • Quick Reference

    TaskCommand
    Generate Ed25519 key<code class="inline-code">ssh-keygen -t ed25519 -C &quot;email&quot;</code>
    Copy key to server<code class="inline-code">ssh-copy-id -i key.pub user@host</code>
    Add key to agent<code class="inline-code">ssh-add ~/.ssh/id_ed25519</code>
    List agent keys<code class="inline-code">ssh-add -l</code>
    Debug connection<code class="inline-code">ssh -vvv user@host</code>
    Fix permissions<code class="inline-code">chmod 700 ~/.ssh &amp;&amp; chmod 600 ~/.ssh/id_*</code>
    Test config<code class="inline-code">sudo sshd -t</code>
    Reload sshd<code class="inline-code">sudo systemctl reload sshd</code>
    Check auth logs<code class="inline-code">journalctl -u sshd --since &quot;5 min ago&quot;</code>
    Agent forwarding<code class="inline-code">ssh -A user@bastion</code>

    Summary

    SSH key authentication is both more secure and more convenient than passwords. Generate Ed25519 keys, use ~/.ssh/config to manage multiple hosts and keys, and always verify file permissions when troubleshooting. The verbose flag (ssh -vvv) tells you exactly where authentication fails.

    ---

    Frequently Asked Questions

    How do I generate an SSH key pair?

    Run ssh-keygen -t ed25519 -C "your@email.com" for the most secure modern option, or ssh-keygen -t rsa -b 4096 for broader compatibility. Save the key to the default location (~/.ssh/id_ed25519) and set a strong passphrase. This creates a private key (keep secret) and public key (.pub file) that you share with servers.

    What is the difference between Ed25519 and RSA SSH keys?

    Ed25519 is a modern elliptic-curve algorithm that's faster, produces shorter keys (68 characters vs 544 for RSA-4096), and is considered more secure against timing attacks. RSA-4096 is the legacy standard with broader compatibility, especially with older systems. Use Ed25519 for all new keys unless you need to connect to systems that don't support it.

    How do I add my SSH key to a remote server?

    Use ssh-copy-id user@hostname which automatically installs your public key in the server's ~/.ssh/authorized_keys. Alternatively, manually append your public key content (cat ~/.ssh/id_ed25519.pub) to the remote server's ~/.ssh/authorized_keys file. Ensure permissions are correct: 700 for .ssh/ directory and 600 for authorized_keys.

    Why is my SSH connection being refused or timing out?

    Connection refused means SSH daemon isn't running or is on a different port — check with systemctl status sshd and verify the port in /etc/ssh/sshd_config. Timeout means a network/firewall issue — verify security groups, iptables rules, and network connectivity. Use ssh -v user@host for verbose output showing where the connection fails.

    How do I use SSH agent forwarding safely?

    Start the SSH agent with eval $(ssh-agent), add your key with ssh-add, then connect with ssh -A user@bastion. This lets you use your local key from the bastion host without copying it there. Only use agent forwarding with trusted servers — a compromised server can use your forwarded agent to access other systems. Prefer ProxyJump (ssh -J bastion target) instead.

    ---

    Related Resources